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Executive Summary 

study Background and Overview 

The 21st Century Community Learning Centers (21st CCLC) program is a federal program 
authorized in 1994 under Title IV, Part B, of the Elementary and Secondary Education Act of 
1965 (ESEA), as amended. The program was originally intended to provide funding to school 
districts to support continuing education and lifelong learning opportunities to children and 
adults to help keep the country’s workforce competitive for the 21st century. The program’s 
authorizing statute was amended by the reauthorization of ESEA in 2002 to provide before- and 
after-school and summer academic enrichment opportunities for children, particularly children 
attending low-performing public schools, and to eligible private school students. The goal of the 
program is to help students meet local and state academic standards in core subjects, such as 
reading and mathematics. In addition to adding to academic content, programs may also provide 
youth development activities, drug and violence prevention, technology education, art and music 
activities, character education, counseling and recreation. The federal program is administered by 
the United States Department of Education, which awards grants to states by formula. States, in 
turn, award subgrants to eligible entities on a competitive basis. Eligible entities include 
education agencies, community-based organizations and other entities operating in either school 
or community settings. 

The previous national evaluation of the 21st Century Community Eearning Centers program 
(Dynarski et ah, 2003; James-Burdumy et ah, 2005), which examined centers funded under the 
pre-2002 legislative authority, found that the centers did not focus on academic achievement and 
had no effects on participants’ academic outcomes. These findings raised questions about the 
level of program quality in after- school programs. The requirements changed to focus more on 
academics, so a study conducted today of program impacts might not find the same results. 

In 2004, the U.S. Department of Education’s Policy and Program Studies Service contracted 
with SRI International and its partner. Policy Studies Associates, to undertake an evaluation of 
the 21st Century Community Eearning Centers The following evaluation questions informed this 
study: 

1. What is the nature of activities in centers that are designed to promote the academic 
development of students? 

2. How do centers vary with respect to regular attendance? 

3. How do center leaders staff their centers, coordinate with other service providers and use 
data to improve programming? 

The sources of data for the study were surveys and site visits. A sample of 516 center 
directors intended to be nationally representative of centers offering academic activities 
completed a written survey in the 2006-07 school year. A subsample of administrators and 
program staff members from 122 centers completed a more in-depth telephone survey on 
attendance and staff characteristics in the same school year. Site visits in fall 2006 and spring 
2007 provided data on the nature of instruction to compare with the survey data; the visits also 
provided observational data on instruction and student participation. The study team interviewed 
program staff members and observed after-school programming at 12 sites (11 served elementary 




school students; one served middle school students). The site visit data are not nationally 
representative and observation protocols used have not been related to any outcomes of 
importance to the program. The study also used grantee- and center-level data collected by the 
21st CCLC program office at the U.S. Department of Education, through the Profile and 
Performance Information Collection System (PPICS) database, to identify the basic center 
characteristics nationwide and to construct the survey and case study samples. 

The study provides descriptive information on the 21st CCLC program; it does not provide 
information on program outcomes or impacts. In addition, the study’s original sampling strategy 
and its reliance on self-reported data from surveys limit the generalizability of the findings and 
provide no basis for making causal inferences. To ensure that all centers surveyed could respond 
to questions about academics, the sampling plan limited the sample to centers that were funded at 
the time of the study, and that offered instruction in reading, mathematics and technology; thus, 
the sample was not nationally representative of all 5,122 centers funded at the time of the study. 
To address this limitation, the data were poststratified to reflect the full population of centers. 
Respondents’ self-reported answers to survey questions may reflect unreliable memory of past 
events and may include responses on academic instruction that center staff considered socially 
desirable. This report presents findings on academic instruction only for centers serving 
elementary school students in which the study team collected observation data. Findings on 
academic instruction for centers serving middle and high school students are reported in 
Appendix A. 

Key Findings 

This report on the 21st Century Community Learning Centers program focuses on the 
implementation of reading and mathematics activities, student attendance and hiring and 
retaining qualified staff in centers from which data were obtained. 

Nature and Quality of Reading and Mathematics Activities 

The program statute requires that centers focus on academics and use research-based 
strategies for instruction. The law requires that students participate in academic activities at a 
frequency that is “sufficient to influence their learning.” 
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According to grantee performance data, nearly all centers funded offered reading^ and 
mathematics activities. 

Ninety-eight percent of all centers funded as of the 2006-07 school year (the time of the 
study) reported that they offered activities in reading, and 94 percent of all centers offered 
activities in mathematics. Whether students are required to participate in these activities, 
however, varies by center. 

Three-quarters of the centers reported that a typical student participated in reading 
activities (75 percent) and mathematics activities (81 percent) for less than 4 hours per 
week. 

Centers serving elementary school students reported that the average student spent the most 
amount of time per week doing homework in a group setting (36 percent) or participating in arts 
or recreation activities (33 percent) (Exhibit ES-1). One-quarter of centers reported that a typical 
student received instruction in reading or practiced reading skills, and 19 percent of centers 
indicated that a typical student engaged in mathematics activities for more than 4 hours per 
week. Thirty-six percent of centers reported that a typical student worked on homework in a 
group setting (which could also include reading and mathematics activities), and 33 percent said 
that the typical student was involved in arts/recreation activities for more than 4 hours per week. 
Because centers were open for about 16 hours per week, on average, student participation for 4 
hours per week in a particular activity represented 25 percent of the available time. 



* Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. 
Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment 
activities but not homework assistance. However, homework assistance activities could include reading activities. 
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Exhibit ES-1 

Percentage of Centers Serving Elementary School Students That Reported Participation 
in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement 

per Week 




-3% 



1% 



1% 



4% 



Hours per Week 
□ More than 4 hours 
2-4 hours 

■ Less than 2 hours 
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0% 20% 40% 60% 

Percent of Centers Reporting 



80% 



1 00% 



Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 5. 
n = 389 

Exhibit reads: Thirty-six percent of centers serving elementary school students reported that a typical student 
participated in homework activities for more than 4 hours per week. 



A majority of centers serving elementary school students reported that reading activities 
included the five essential components of reading instruction. 

Many centers serving elementary school students reported emphasizing comprehension, 
fluency, vocabulary, phonics and phonemic awareness in at least some activities. Seventy-four 
percent of centers serving elementary school students reported that they focused on 
comprehension in all or most instructional activities in reading, compared with 52 percent that 
concentrated on phonics skills in all or most activities. Observational data were consistent with 
the pattern reported by centers serving elementary school students: 86 percent of observed 
activities focused on comprehension, compared with 46 percent that focused on phonics. 
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However, the observation data were collected from a small number of purposively sampled 
sites. ^ Thus the comparison may not be useful. 

Centers reported that mathematics activities focused on basic skills. 

Seventy-one percent of centers surveyed serving elementary school students reported 
stressing operations with whole numbers in all or most mathematics activities. Centers serving 
elementary school students also reported that they were more likely to engage students in tasks 
that required simple rather than complex problem solving. Sixty-eight percent of centers reported 
asking students to practice basic facts in all or most instructional activities in mathematics. 
Observational data from a small, purposive sample of case study sites were consistent with the 
pattern of emphasizing basic mathematics facts: 83 percent of observed activities involved 
practice with basic facts. 

Observers in case study sites found that staff providing instruction used active teaching 
strategies in academic activities, communicating goals clearly to students in most activities. 

However, staff were more likely to use multiple teaching strategies in reading than in 
mathematics. Staff providing instruction communicated the goals, purposes and expectations of 
activities to students more often in mathematics activities (89 percent) than in reading activities 
(60 percent). Staff providing instruction used multiple strategies in 22 percent of the mathematics 
activities observed, compared with 53 percent of the reading activities observed. 

Student Attendance in Center Activities 

Researchers have linked regular participation to better outcomes for students in after-school 
programs (e.g., Lauer et ah, 2006). Although ED’s annual performance reporting guidelines 
define regular attendance as 30 days or more per year, the number of days required to have an 
effect on academic achievement is not known. 

Centers reported that about half of their students attended roughly 2 days a week or more. 

The study team asked a random subset of 140 centers in the study to report on student 
attendance and participation. Just 75 of the 119 centers (63 percent) that completed surveys 
indicated they could track these data. The centers that could track attendance indicated that 44 
percent of all center students attended 60 days (roughly 2 days per week) or more in the last year. 



^ It is important to note that the study team asked center directors to obtain these data from someone familiar with 
the activities, but they may have responded without consulting an expert. Additionally, center directors may have 
had reasons to provide socially desirable responses to a U.S. Department of Education (ED) survey. 
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More than half of all centers reported having policies that required students to attend at 
least 2 days a week, hut attendance policies were not associated with greater attendance. 

Centers’ attendance policies varied. About half of all centers (56 percent) had policies 
requiring student attendance at least 2 to 3 days per week in order to remain enrolled in the after- 
school program. Thirty-eight percent of centers reported requiring attendance daily. Twenty-six 
percent did not require attendance. There were no significant differences in attendance 
requirements between school-based and nonschool-based centers. A higher percentage of centers 
serving elementary school students (41 percent) were more likely to have policies requiring 
attendance than those centers not serving this age group. In contrast, 31 percent of centers 
serving middle school students and 22 percent of centers serving high school students required 
attendance every day. Beyond basic attendance requirements, more than half of the centers 
required the participation of all students in specific academic activities, including homework 
help (73 percent), reading activities (60 percent), mathematics activities (58 percent) and tutoring 
(14 percent). 

In centers that required attendance, 66 percent of students attended for more than 30 days; 

62 percent of students attended for the same duration in centers that did not require attendance. 
No relationship was found between any kind of attendance policy and actual attendance for 
centers overall or for centers serving elementary school students. For the middle grades, centers 
that required attendance every day had higher attendance than those that did not require 
attendance. In high school, centers that required attendance 2 to 3 days per week had higher 
attendance than those that did not require attendance. 

Although attendance rates varied little by center type, elementary school students were 
more likely to attend center programming than older students. 

The pattern of higher attendance for elementary-serving centers than for secondary-serving 
centers mirrors that of the previous national evaluation. 

Centers that served elementary school students and had adequate tracking systems reported 
that 48 percent of students attended 60 days or more in the 2005-06 school year, or roughly 2 
days per week. Centers serving the middle grades indicated that 36 percent of students attended 
this often, and centers serving high school students cited 30 percent of their students’ meeting 
this attendance level. Researchers conducting the previous national evaluation found a similar 
pattern of results for elementary school students but not for middle school students: In their 
study, 55 percent of elementary students attended 51 or more days, but just 20 percent of middle 
school students attended that often (Dynarski et al., 2004). Their study used different methods to 
study attendance, however. 



^ Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations 
of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a 
separate question about consequences of nonattendance. “Require attendance” may mean that center directors have a 
rule that only students who attend five days per week may enroll in the program; in other instances, requiring 
attendance could simply mean to center directors that there is a norm that students will attend regularly. 
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Organizational Supports for Instructional Quality 

Offering high-quality instruction in reading and mathematics requires recruiting, developing 
and retaining high-quality staff, as well as developing policies and programs that attract and 
retain students and reflect students’ academic needs. 

Centers reported that they relied primarily on part-time staff, who were unlikely to receive 
benefits from their work in the center. This latter finding is not surprising, as the centers 
are open for an average of only 16 hours per week. 

Seventy-six percent of program staff members in centers who led instructional activities 
reported working fewer than 20 hours per week. Centers infrequently offered job benefits for 
part-time staff members. Fourteen percent of centers reported offering a retirement savings plan 
to part-time staff, 1 1 percent offered paid time off for vacation and sick leave, 8 percent offered 
health insurance and 4 percent offered tuition reimbursement. It is important to note that part- 
time staff may include individuals who have full-time teaching positions in addition to working 
at the 21st CCLC center. 

To provide professional development to staff, about half of centers reported offering 
opportunities through training courses or conferences. 

Centers indicated that the school-day teachers they employed may have had increased 
opportunities for staff development through activities offered by their districts and schools than 
did other types of center employees. Sixty-two percent of centers offered other paid training or 
professional development to full-time staff, and 55 percent did so for part-time staff. Fifty-three 
percent of centers offered paid conference attendance to full-time staff; 39 percent offered this 
opportunity to part-time staff. In the case studies, center directors in school-based programs said 
they relied on professional development opportunities the staff received through the district. 

They also reported that they encouraged staff to share their ideas for innovative programming 
through weekly staff meetings that served as school-based professional development 
opportunities. 

Across all centers, 29 percent of staff had worked at the center for less than 1 year. 

About half of centers (48 percent) reported that the primary reason for staff turnover was 
graduation from school or completion of a program of study. Other commonly reported reasons 
for staff turnover were lack of benefits and the centers’ inability to offer full-time positions. 

Centers reported that more than two-thirds of staff providing instruction in reading and 
mathematics had prior experience as certified classroom teachers or as instructional 
specialists in reading or mathematics. 

Fifty-five percent of the staff who provided instruction in reading or mathematics had been or 
were, at the time of the study, regular classroom teachers, and 23 percent had been instructional 
specialists in reading or mathematics. Twenty-three percent of staff providing instruction in 
reading or mathematics were currently or had served as classroom aides. 
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Centers reported that nearly two-thirds of reading and mathematics instructors had a 
bachelor’s degree or higher (64 percent for reading and 63 percent for mathematics). 

Centers reported that a little more than a third of their instructors reported having only a 
bachelor’s degree (32 percent for reading and 33 percent for mathematics). 

A majority of centers reported using assessment data to improve existing program 
offerings and evaluate program success. 

Seventy-one percent of centers reported having access to whole-school state assessment data. 
In addition, more than four-fifths reported that they received state assessment results for 
individual students at their centers (83 percent in reading and 82 percent in mathematics). 

Centers said that they used a variety of data to assess student academic growth, make program 
adjustments or evaluate program success periodically. Almost half (47 percent) of centers noted 
that once or twice a year they used results from tests administered at the students’ school, while 
34 percent of centers said that written reports from students’ teachers were used to assess 
academic growth once or twice annually. 

About 40 percent of centers reported some involvement with supplemental educational 
services (SES). Like 21st Century Community Learning Centers, SES is intended to 
provide after-school academic activities to students. A small percentage (9 percent) of 
centers said they coordinated their activities with SES providers. Fifteen percent of centers 
reported being authorized to provide SES themselves. 

On average, each center that was an SES provider reported offering supplemental 
instructional services to 38 students in reading and mathematics. Just 9 percent of all centers 
reported coordinating with one or two providers, and only 5 percent of all centers reported that 
their coordination activities focused on aligning schedules with the providers, while 4 percent 
indicated they coordinated their academic support activities with the providers. 

Just under one-third of all centers reported that coordination with staff from the school- 
day instructional program was a challenge to implementing high-quality programming. 
However, the percentage was higher for nonschool-based centers. 

Lack of information about students’ academic needs, school-day teachers’ lack of 
responsiveness to requests from after-school staff for information and lack of information about 
the school-day curriculum were cited as barriers to implementing high-quality programming for 
22 percent to 32 percent of all centers. Barriers to obtaining information about student needs 
were greater for nonschool-based centers than for school-based centers. Thirty-six percent of 
nonschool-based centers reported lack of information as a barrier, compared to 20 percent of 
school-based centers. Barriers were also greater for nonschool-based centers with respect to 
responsiveness of school staff: 39 percent of nonschool-based centers identified lack of 
responsiveness as a barrier, compared to 29 percent of school-based centers. 
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Conclusions 

Survey data and site observations indicate that 21st Century Community Learning Centers 
focused on reading and mathematics enrichment. Compared with the breadth of reading skills 
emphasized, mathematics instruction covered a narrower range of basic skills. Centers reported 
that 44 percent of students attended 60 days or more in the last year, which amounts to roughly 2 
days per week. Elementary school students attended after-school activities for more days in the 
school year than did middle and high school students. About half of the centers reported using 
data for a variety of purposes, including program evaluation and ongoing review of programming 
activities. The majority of centers reported having access to state assessment data results on 
individual students and many reported using this and other information to inform program 
practice. 

Although there were few differences between school-based and nonschool-based centers, 
school-based centers were more likely to report emphasizing higher-order skills such as asking 
students to make predictions about something they were reading and talking or writing about 
answers to questions related to something they had read. At the same time, nonschool-based 
centers had students read teacher-selected books more often than school-based centers did. 

The findings of this report suggest three challenges that centers face in implementing their 
programs: (a) staff departures after graduating from school or completing a program of study, as 
the lack of benefits makes it difficult to retain high-quality staff; (b) a lack of up-to-date 
information about students’ individual needs and (c) low attendance rates, the remedy for which 
requires more than simply having attendance policies. 

The study did not directly measure instructors’ knowledge or skills, but future studies could 
examine detailed measures of instructor knowledge to predict differences in instructional quality. 
Future studies could also examine the content of professional development provided for center 
staff. Finally, future work that examines the quality of reading programming could focus on how 
best to capture information on instruction in after-school programs since this study had 
challenges in this area. Future studies could also identify effective practices for improving 
attendance levels among participants. 
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1. study Background, Goals and Methods 

Since the reauthorization of the Elementary and Secondary Education Act (ESEA) of 2002, a 
stronger emphasis has been placed on out-of-school learning to improve student academic 
achievement. As policy makers, schools and districts contemplate possible interventions, one 
option is to increase the amount of time spent on learning, such as through before- and after- 
school programs. Additional hours provide more opportunities for instruction, academic 
assistance and enrichment or experiential activities to support learning. 

The body of rigorous research on the effects of extended learning time on student academic 
achievement is not extensive, but it suggests the potential for a positive effect on academic 
performance (e.g., Lauer et ah, 2006; Zimmer et ah, 2007). How the additional time is used is 
what matters. For example, a review of the literature on time and learning found that there was 
no relationship between allocated time and student academic achievement, some relationship 
between engaged time and achievement and a greater relationship between time spent on 
academic learning and achievement (Aronson, Zimmerman, and Carlos, 1999). Large-scale 
survey studies find a similar relationship between time spent on academics in specific content 
areas and academic achievement in those areas (Rowan, Correnti, and Miller, 2002). 

The 21st Century Community Learning Centers (21st CCLC) program is an example of a 
federal effort to improve academic achievement by extending learning opportunities. The 
program, which is funded by the U.S. Department of Education and administered by the states, 
seeks to provide opportunities for academic enrichment, youth development and other activities 
for students before and after school, as well as during the summer, and to offer the families of the 
students served opportunities for literacy and related educational development. Past program 
evaluations have generated questions about the impact of federally funded after-school services 
on student academic achievement. In the previous 2-year national evaluation study of the 21st 
CCLC program, which was conducted early in the program’s history, researchers found that 
attendance was irregular in many centers. In addition, the evaluators reported that academic 
opportunities in core subject areas such as reading and mathematics were limited and that 
participation in the program had no effect on student academic achievement (Dynarski et ah, 
2003; Dynarski et ah, 2004; James-Burdumy et ah, 2005). 

The current study examined characteristics of programming, building on the findings from 
the earlier evaluation. In 2004, the U.S. Department of Education’s Policy and Program Studies 
Service contracted with SRI International and its partner. Policy Studies Associates, to evaluate 
the programming that the 21st CCEC program offers students and their families. One purpose of 
the evaluation was to provide evidence regarding how 21st CCEC program grantees, center 
directors and center staff implemented activities — in particular, academic activities focused on 
reading and mathematics. In addition, the study examined organizational supports that previous 
research suggests may be associated with high-quality programs as indicated, for example, by the 
hiring and retention of staff with experience and expertise relevant to their roles in the centers. 
This report presents the findings of the evaluation. 




The 21st Century Community Learning Centers Program 

The Elementary and Secondary Education Act of 1965 (ESEA), as amended by the 
Improving America’s Schools Act of 1994 (P.L. 103-382), initially authorized the 21st CCLC 
program as a small discretionary grant program. In the authorizing legislation, Congress 
described local public schools — particularly those in rural areas and inner cities — as valued 
institutions to which communities already turned for educational and other essential services; 
thus, funding would go to school districts to develop centers. Centers were to provide continuing 
education and lifelong learning opportunities “to individuals of all ages” and were to help keep 
the country’s workforce “competitive and successful ... [in] the high technology, global 
economy of the 21st century.” Each center was required to offer no fewer than 4 of 13 authorized 
activities, which included: services for school-age students; senior citizen programs; children’s 
day care services; and employment counseling, training and placement for adults. In 1998, an 
absolute priority was added for “activities that offer expanded learning opportunities for children 
and youth in the community.” 

The 21st CCLC program underwent a significant transformation when Congress reauthorized 
it in January 2002 as Title IV, Part B, of the ESEA (P.L. 107-110). The reauthorization 
emphasized improving academic achievement for low-performing schools with high percentages 
of low-income students. 

The reauthorization also established a new set of authorized activities that centers could 
offer: 



• Remedial education activities and academic enrichment learning programs, including 
providing additional assistance to students to improve their academic achievement 

• Mathematics and science education activities 

• Arts and music education activities 

• Entrepreneurial education programs 

• Tutoring services (including those provided by senior citizen volunteers) and mentoring 
programs 

• Activities that emphasized language skills and academic achievement for students with 
limited English proficiency 

• Recreational activities 

• Telecommunications and technology education programs 

• Expanded library service hours 

• Programs that promoted parental involvement and family literacy 

• Programs that provided assistance to students who had been truant, suspended or 
expelled, to allow them to improve their academic achievement 

• Drug- and violence-prevention programs, counseling programs and character education 
programs 
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As significant as these changes to the goals and authorized activities were, changes relating 
to program administration following reauthorization were at least as important. Congress 
transferred the responsibility for administering the program from the U.S. Department of 
Education to each state when funding rose to $1 billion in fiscal year (FY) 2002. The U.S. 
Department of Education currently awards grants to states using a formula based on a state’s 
allocation under Title I, Part A, of the ESEA, which takes into account the state’s size and the 
proportion of students from low-income families. Focal education agencies and a variety of 
community-based organizations, including for-profit businesses, nonschool-based organizations 
and faith-based organizations, are eligible to apply for subgrants from the state. Each state is 
required to submit an application for implementing the program, and the U.S. Department of 
Education is responsible for approving those applications. States then operate their own grant 
competitions, according to their approved applications, in which they determine the duration of 
program grants (between 3 and 5 years) and specify their own requirements for matching funds 
(up to a one-to-one maximum). 

Individual programs also have flexibility under the reauthorized program. States can allow 
programs (subgrantees) to apply to implement programs in various locations — for example, 
choosing to operate in elementary or middle school buildings, or in a location more accessible to 
students and their families. This policy change made it more likely that nonschool-based private 
and public entities could operate with support from the 21st CCEC program. In addition, 
applicants for subgrants have some flexibility in setting program goals. As part of their grant 
applications, subgrantees must identify the community need for their program and base their 
offerings on the need identified. As principal grantees, states are also required to select program- 
and student-level outcomes. 

Although the federal guidelines specify a range of measures for which grantees must report 
data, states have considerable flexibility in determining how to hold local programs accountable 
for results. Certain states allow local programs to set program goals and measures within the 
bounds of federal and state priorities. Centers are required to report data on the Government 
Performance Results Act measures for the program through the Profile and Performance 
Information Collection System (PPICS) system. They are given flexibility on whether they 
report on grades or state assessment results, but they are required to report on one or the other. 

To monitor program progress, the U.S. Department of Education contracted with Eeaming 
Point Associates in October 2003 to develop the PPICS. This online data collection and reporting 
system gathers grantee- and center-level data aligned with overall program goals to enable local 
programs to report on progress toward the outcomes they set out to achieve as part of their 
grants. This study used the PPICS database to identify basic characteristics of all centers to 
construct the survey and case study samples. In addition, PPICS was the source of contextual 
information for the survey data and the case study sites. 

In FY 2009, Congress appropriated more than $1.1 billion for the 21st CCEC program. 
Currently, the federal program supports 1,585 local programs within more than 9,500 centers in 
53 states and U.S. territories. Roughly two-thirds of all centers serve students in each of the 
elementary grades; slightly fewer than half serve students in grades 6, 7 and 8 and one in six 
centers serve students in the high school grades. Exhibit 1-1 summarizes basic information about 
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funded centers that were in the population at the time of this study, i.e. 2005-06, and about 
centers targeted by the study. 



Exhibit 1-1 

Characteristics and Type of and Grade Levels Served by Funded and Sampled Centers 



Characteristic 


Funded Centers 


Centers in Sampie 


Located in an urbanized area 


74% 


74% 


Minority population in area served 


37% 


36% 


Center Type 






Located in a school 


92% 


71% 


Located in a nonschool setting 


8% 


29%* 


Grade Levei Served 






K 


45% 


45% 


1 


57% 


57% 


2 


60% 


60% 


3 


66% 


64% 


4 


67% 


64% 


5 


68% 


64% 


6 


54% 


54% 


7 


39% 


41% 


8 


38% 


41% 


9 


13% 


41% 


10 


11% 


13% 


11 


11% 


12% 


12 


11% 


12% 



* For this study, nonschool-based centers are overrepresented to facilitate the 
examination of differences between nonschool-based centers and school-based centers. 

Source; Profile and Performance Information Collection System database. 
n= 5,122 for funded; n = 516 for sample 

Exhibit reads: Seventy-four percent of centers are located in urbanized areas. 

Background 

This study attempts to analyze the extent to which activities funded by the 21st CCLC 
program have characteristics known to be associated with positive student academic 
achievement. Throughout much of their history, after-school programs have been defined by 
their diversity of organizational sponsors, voluntary youth participation, modest levels of adult 
direction and commitment to meeting the social and developmental needs of low-income youth 
(Halpem, 2002; Muller and Frisco, 1998). Over time, however, programs have shifted away from 
voluntary and toward mandatory participation, and they have increased the level of adult 
direction, especially with respect to homework help (Halpem, 2002). The pressure on schools to 
meet the academic needs of disadvantaged youth has also led to increased emphasis on 
academics in after-school programs (Brown, 1999). 

This study focuses narrowly on academic activities in two areas, reading and mathematics, in 
after-school programs and does not address practices associated with outcomes other than 
academic achievement. This narrow focus is significant, since research syntheses of 
experimental and quasi-experimental research studies suggest that after-school programs may 
produce positive outcomes in areas other than student academic achievement (Durlak and 
Weissberg, 2007; Lauer et ah, 2006). The authors of these syntheses also conclude that programs 
do not have to be exclusively academic in focus to have positive impacts on academic 
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achievement. This study focuses on reading and mathematics activities, however, because the 
2004 program evaluation identified these areas as ones where programs might be strengthened. 

Evidence has shown that programs that take place outside the school day can help improve 
reading and mathematics achievement. Lauer and colleagues (2006) conducted a meta-analysis 
of studies that examined the effectiveness of after-school and summer strategies for improving 
achievement in reading and mathematics. Their study, conducted under contract for the U.S. 
Department of Education’s Institute of Education Sciences, was limited to studies that measured 
achievement in reading or mathematics and used comparison-group designs. They found small 
but significant positive effects of programs on achievement in both reading and mathematics, and 
larger positive effect sizes for programs with specific characteristics such as tutoring in reading. 

Conceptual Framework for the Study 

Because the enhanced focus on academics is relatively new in the field of after-school 
programming, few studies have been conducted of the program quality of after-school academic 
instruction. Until now, studies have principally relied on frameworks for studying youth 
development in community settings (e.g., Eeffert, Benson, and Roehlkepartain, 1997; National 
Research Council & Institute of Medicine, 2002), rather than on frameworks that focused on 
specific academic content or instructional strategies. The framework that guided this study, 
however, drew on both research conducted in community settings and research conducted in 
schools about instructional quality. In this respect, the study framework was similar to Huang’s 
(2001) system of program quality indicators for middle and high school after-school programs, 
which were based on research carried out in both schools and after-school programs (see also 
Eittle, 2007). 

Each subsequent chapter first briefly describes the relevant components of the framework 
that guided the study (see Exhibit 1-2) and the research that supports the inclusion of each 
component of the framework. The chapters then set forth relevant study findings. It is important 
to note that while this study is informed by the broader universe of after-school programming, it 
is focused on only the 21st Century Community Eeaming Centers program, and its findings are 
not intended to be applied to other after-school programs. 

The framework below helped organize the presentations of findings, but it has some 
limitations. Its principal limitation is its narrow focus on those elements needed to promote 
participation in reading and mathematics instructional activities in 21st CCEC programs, which 
were the focus of the study. The framework is by no means intended to be inclusive of the broad 
range of elements of program quality that may be needed to improve academic achievement or 
related school outcomes, such as attendance. Nor does the framework encompass all research 
relevant to judging program quality in after-school programs. Research about which program 
elements contribute to social and emotional outcomes for youth, for example, is excluded from 
the framework because those findings fall outside the focus of the study. 
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Exhibit 1-2 

Framework for the Study: Supports for Student Participation in Quality Academic 
Instruction That Leads to Increased Achievement 



INSTRUCTIONAL 

SUPPORTS 




Exhibit reads: In the framework for the study, student participation in quality academic instruction is 
hypothesized to lead to desired student outcomes for the program. 



Chapter addresses the components of high-quality academic instruction, which is one of 
the most important influences on academic achievement (Cohen, Raudenbush, and Ball, 2003): 

• Sufficient exposure to academics after school on a consistent basis. The frequency and 
intensity of student participation in academic activities at centers needs to be sufficient to 
influence their learning. 

• Focus on core academic content. Research in reading has identified core skills (e.g., 
phonemic awareness) that are essential to literacy. Likewise, research in mathematics has 
identified several content strands that build students’ mathematical proficiency (e.g., 
peer-assisted learning strategies, solving cognitively complex problems, using computers 
and graphing calculators to solve problems). 



The information presented in this chapter is limited to the study’s elementary school findings. Data from programs 
serving middle and high school students are presented separately in Appendix A. Comparisons between school- 
based and nonschool-based centers are based on elementary-level findings. 
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• Guidance from instructors with education and experience related to their roles. 
Instructors with backgrounds in helping students in reading and mathematics are likely to 
be better prepared to offer high-quality instruction matched to student needs in these 
domains. Staff members’ active guidance of student learning is likely to be more 
effective than passive supervision of students. 

• Feedback to students and parents on academic progress. Feedback to students helps 
develop their metacognitive skills and motivates improvement. Parent feedback supports 
caregiver involvement in student learning, a key component of academic success. 

Chapter 3 addresses the components of student participation necessary to affect academic 
achievement outcomes: 

• Regular attendance. Regular center attendance is necessary for students to benefit from 
after-school instruction. 

• Student engagement in academic activities. In addition to attendance, being on task and 
focused on academics when present support students’ motivation to learn and to 
appreciate the importance of developing proficiency in reading and mathematics. 

Chapter 4 addresses the necessary supports for high-quality instruction: 

• Center staffing and policies. Centers are responsible for providing pay and benefits at 
levels that help attract and retain qualified staff, and for providing opportunities for staff 
to develop and enhance skills relevant to their roles. 

• Alignment of activities to student needs. High-quality instruction is aligned to student 
needs. To understand student needs, after-school programs need to be well coordinated 
with school-day activities and use data from student assessments (or conduct their own 
student assessments) to adjust program offerings accordingly. 

• Adult support. After-school programs provide an opportunity to extend students’ learning 
beyond the school day, but the effectiveness of that help depends on the quality of the 
relationships between adults and students. 

Study Design and Data Collection Methods 

Evaluation Questions 

This evaluation of program quality addressed three main questions: 

1 . What is the nature of activities in centers that are designed to promote academic 
development of students? 

2. How do centers differ with respect to regular attendance? 

3. How do center leaders staff their centers, coordinate with other service providers and 
allocate their resources to provide quality programming? 
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Study Design 

The evaluation included the following components: (a) a national survey of center directors, 
sent to a sample of 600 centers in 2006-07, (b) a focused telephone survey of center directors 
conducted among a sub sample of 140 of those centers in 2006-07 and (c) case study visits to 12 
sites in fall 2006 and spring 2007. The focused telephone survey sample was nested within the 
survey sample, which was nationally representative of centers offering instruction in reading, 
mathematics, and technology (see explanation below on weighting). Approximately 70 percent 
of the sample consisted of school-based sites. Nonschool-based sites were deliberately 
overrepresented to enable statistical comparison with school-based sites. ^ 

The following outline briefly describes the sources of data, data collection procedures and 
response rates, along with a brief explanation of how to interpret the data presented in the 
exhibits. (See Appendix C for information about the selection of samples for the national survey, 
focused survey and case studies.) 

Data Sources and Data Collection Procedures 

National survey. The survey provided data from a nationally representative sample of 600 
center directors to whom surveys were sent, of whom 516 responded. The survey elicited 
information regarding centers’ objectives and approaches and basic staffing information, as well 
as the indicators that they use to measure success, their methods for tracking outcomes and their 
methods for communicating with parents about student progress. In addition, the survey 
collected data on the organization of activities aimed at improving students’ reading and 
mathematics skills. In selecting centers, we did not distinguish among centers offering summer, 
before- or after- school programming. 

The surveys were mailed in September 2006 to the center director in charge of after-school 
services at each of the 600 selected centers. Survey responses were accepted until January 19, 
2007. After eliminating centers that could not be reached or were ineligible for the study (e.g., 
because funding had expired), the size of the sample for the national survey was 558. Directors 
of 516 centers completed the national survey, a response rate of 92.5 percent. 

Focused telephone survey. To provide more in-depth information on program quality, a 
focused telephone survey collected data from a nationally representative subsample of 140 center 
directors from the 600 centers surveyed by mail. The telephone survey gathered information 
about the extent to which centers tracked students’ level of proficiency on state tests in reading 
and mathematics, the targeting of services and centers’ coordination with local supplemental 
educational services providers. Questions were also asked about student attendance and the 
qualifications of center directors and staff. To collect data on staff qualifications, each center 
director used a method provided by the research team to randomly select three staff members. 

All centers were asked to provide data on the chosen staff members. If centers had fewer than 
three staff members, directors completed information regarding all staff members. Printed 
versions of the focused telephone surveys were mailed, along with the national surveys, to the 



^ School-based sites are not always administered by schools or school districts and nonschool-based sites are not 
necessarily administered by a community-based or nonprofit organization. Of the 516 schools in our final sample, 
nonschool grantees administered 210 programs (101 in school settings; 109 in nonschool settings) and schools or 
school districts administered 306 programs (266 in school settings; 40 in nonschool settings). 
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140 selected centers, and respondents had the option of mailing in their completed surveys or 
communicating their responses by telephone during the data collection period. Survey responses 
were accepted until January 19, 2007. After eliminating ineligible and unreachable centers, 132 
centers were chosen for the telephone survey. Center directors completed 122 focused telephone 
surveys — a response rate of 92.4 percent. 

Site visits. The study team visited 12 sites between fall 2006 and March 2007. Eleven of the 
site visits were to elementary schools; the 12th was to a middle school site. Two sites were 
visited in each of California, North Carolina, South Dakota and Maryland, and four sites were 
visited in Florida. To ensure comparability of data collected across sites, all site visitors received 
training at the beginning of the study. Two researchers then visited each site, where they 
conducted program observations, document review and interviews with center directors, staff 
and school-day instructors. After the site visits, each pair of researchers completed a debriefing 
form to ensure that they had captured comparable data aligned with the research questions. 

Site visit protocols included interviews with grant directors, center directors, program staff 
and school personnel. The protocols also included a structured observation of academic and 
enrichment activities. The case studies provided additional evidence about the nature of 
academic instruction and about how well center practices aligned with policy goals for the 
program and with previous research on program quality. The observation form used in the study 
was adapted from an earlier study Policy Studies Associates conducted of after-school programs 
funded by The After School Corporation (TASC) (Birmingham, et ah, 2005). That instrument 
included three constructs used in this study: active teaching strategies, teacher relationships and 
engagement. Four items comprised the active teaching strategies construct: communicating 
goals, using multiple teaching strategies, challenging students to move beyond current skill 
levels and asking students to expand on answers and ideas. For the teacher relationships 
construct, there were also four items: using positive behavior management techniques, showing 
positive affect toward students, attentively listening to and/or observing students and 
encouraging students to share ideas, opinions and concerns. Student engagement had one item. 
For each observation, the study team rated each of these items on a four-point scale, ranging 
from “did not occur” to “highly characteristic.” A concurrent validity study of this instrument 
conducted in 2006 found ratings on these constructs to be significantly correlated with student 
ratings of settings observed by researchers (Pechman, et ah, 2008). Each item is reported 
separately but in a group with other items in the constructs identified in the earlier research; no 
weights were assigned to the items. The observation instrument can be found in Appendix F. 

Because this study focused on academic instruction, the study team added content to the 
observation form related to reading and mathematics (in Appendix F). The items added focus on 
reading skills and activities and mathematics skills and activities asked about in the national 
survey. For each activity, observers rated whether there was an explicit focus on the skills and 
activities, an implicit focus or no focus. The distinction between explicit and implicit focus is an 
important one for some after-school programs, which may make academic goals implicit in an 
effort to attract participants and increase engagement (Vadeboncoeur, 2006). A limitation of 
these observational constructs is that separate validity data are not available, as this was the first 
time the constructs were used. 
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SRI provided training to all case study researchers in the instrument, and researchers 
conducted joint observations in the field to establish the reliability of the instrument. The training 
session provided guidance on the definitions of categories used in the instrument; participants 
also reviewed a codebook with examples of types of activities to help them learn how to use it. A 
total of 13 joint observations took place on site visits. For the active teaching strategies, raters 
agreed on the exact rating in 64 percent of observations and were within one point on the four- 
point scale 94 percent of the time. For on-task ratings, raters agreed on an exact rating 69 percent 
of the time, and 100 percent of the ratings were within one point. Agreement was lower for 
reading skills (63 percent exact agreement) and activities (65 percent), but higher for 
mathematics content (96 percent) and skills (97 percent). 

Weighting 

All the survey data were weighted to reflect the full population of eligible centers and to take 
into account nonresponse. Centers included in the original sampling frame for this study had to 
meet all of the following four criteria: (a) receiving funding at the time the sample was drawn; 

(b) offering reading activities; (c) offering mathematics activities and (d) offering technology- 
related activities.^ Nonschool-based centers were deliberately oversampled and were weighted to 
reflect their percentage in the population of funded centers through the variables discussed 
below. To arrive at conclusions about the national population of centers, the data were 
poststratified to reflect the full population of funded centers by using the variables discussed 
below. The study team also used Deming’s procedure to weight the respondents so that the 
weighted marginal counts were the same as in the universe of sites funded at the time of the 
study and in operation for at least a year. (For example, the weighted number of responding 
elementary schools was the same as the number of elementary schools in the universe.) 

The specific characteristics used to poststratify the respondents came from the PPICS 
database and from U.S. census data on characteristics of persons living in ZIP codes where 
centers were located. The variables from the PPICS database were the numbers of students 
served, program longevity, centers’ history of offering academic services, primary grade levels 
served and number of weeks per year centers were open. Data from U.S. census ZIP code files 
were the percentage of the ZIP codes’ population living in an urbanized area and the percentage 
of minority population in the area. 

Limitations of the Study 

One limitation of the study pertains to the representativeness of the sample. Because of the 
original emphasis on the quality of instructional services in reading, mathematics and 
technology, the study team excluded sites that did not offer all those services. Although nearly 
all sites offered reading and mathematics activities, 19 percent of centers did not offer 
technology services and thus were excluded from the sample. 

The specific characteristics used in poststratifying the respondents, using the PPICS database 
and U.S. census data from ZIP codes where centers were located, equalized the weighted sample 
and the universe of centers in regard to all characteristics expected to influence survey responses. 
The reweighting thus essentially eliminated the biases caused by these variables. A potential 



® Centers provided self-reported answers to these questions through the PPICS database. 
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limitation is that reweighting did not eliminate biases not associated with these variables, nor did 
it eliminate biases that were functions of variable interactions, such as between the size of the 
community served and a center’s academic offerings. 

A second limitation was the study’s heavy reliance on self-reported data. Self-reported data 
on instruction can be inaccurate, and observation data do not always confirm what instructors 
report they are doing in their classrooms (Burstein et al., 1995). One source of bias is instructors’ 
memory, which has been found to be poor for instructional events that are rare or occur 
infrequently (Hoppe et al., 2000; Rowan, Camburn, and Correnti, 2004; Sudman and Bradburn, 
1982). Response choices can also influence survey respondents’ answers, either by cueing 
respondents to supply socially desirable answers or by suggesting estimation strategies that 
systematically bias the obtained reports (Schwartz, 1999; Schwartz and Oyserman, 2001). 
Questions about frequency of events or the number of hours spent in certain academic activities 
may be particularly biased, because respondents tend to answer these questions quickly at a cost 
of accuracy (Burton and Blair, 1991; Marquis, Marquis, and Polich, 1986). 

The study team sought to reduce the impact of self-report bias by using established 
techniques. The questionnaire asked center directors to ask their instructors about specific 
pedagogies and behaviors, a strategy known to increase the reliability of teacher self-reported 
data (Caret, et al., 2001; Herman, Klein and Abedi, 2000; Koziol and Burns, 1986; Mullens, 
1998; Ross et al., 1997). In addition, respondents were asked to skip items if earlier items on the 
survey established that respondents could not report accurately. If, for example, respondents to 
the telephone survey reported that they did not keep accurate records of the number of 
individuals who attended in the past week, they were asked not to report on the number of 
students who attended 90 days or more in the past year. This approach resulted in some 
additional missing data, but it likely reduced the bias in the responses obtained. 

When possible, the team checked the consistency of reporting across questions and against 
data reported by centers in their annual reports. For example, centers provided information about 
their operating hours in annual reports, and they also answered questions about the number of 
hours typical students participated per week. Despite these precautions, bias may still have 
affected the responses to some items, particularly those related to academic instruction. Center 
directors who were mailed the questionnaire may not have asked their instructors to complete it 
as instructed, but instead may have completed the surveys themselves. Center directors, given the 
roles they play in centers, may be reliable informants about record keeping, data use and 
organizational supports for program quality. They rarely provide instruction to students, 
however, so they may not have been good sources for information about instruction. Because we 
were able to compare survey results on instruction with observation results for the centers 
serving elementary school students, we have focused in Chapter 2 of this report on findings for 
centers serving elementary students. Descriptions of academic instruction for centers serving 
middle and high school students appear in Appendix A. Appendix E provides detailed statistics 
for data presented in the tables in Chapter 2. 

A third limitation is that the study team did not distinguish among before-, after- and summer 
school programs. This limitation is important, because a review by Lauer and colleagues (2006) 
found that summer programs were more likely to emphasize academics than programs offered 
during the regular school year. Because this study does not distinguish among different types of 
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programs, the data collected in it may be of limited utility in addressing concerns about 
attendance raised within the 21st CCLC program. 

There are also three important limitations of the observation data. First, the lower agreement 
among ratings for reading skills and activity implies these data are less reliable than data 
associated with the other ratings. Raters had difficulty agreeing on which skills were explicit and 
which ones were implicit; thus, ratings of reading activities were measured with a higher level of 
error than other ratings and should be interpreted with caution. Second, observations were 
purposively sampled, so the data collected are not nationally representative and observation 
protocols used have not been related to any outcomes of importance to the program. We initially 
sampled sites for the study using the promising practices framework (Luce and Thompson 2005). 
The evaluation team analyzed the data from the case studies with the intent of comparing 
promising versus typical sites along such dimensions as center goals, staffing and professional 
development; attendance patterns and policies; academic program offerings and resource 
management. However, the analysis yielded few meaningful differences between typical and 
promising centers; therefore, the focus of data analysis shifted to examining consistency between 
observation and self-reported survey data. Because the observations were few and purposively 
sampled, we cannot draw inferences from these data about program practices nationwide. Data 
from the observations can be used, however, to qualitatively examine whether the patterns are 
similar between the observation and survey data. Descriptions of observation ratings for site 
visits appear in Appendix B. A third limitation is that although analyses were conducted that 
demonstrated the consistency of the OST scales, no analyses have been conducted to establish 
the predictive validity of the scales (i.e., there is no evidence about the relationship between the 
measure and outcomes of interest such as youth behavior and academic outcome measures). 

Presentation of Survey Data in the Report 

Throughout the report, differences among different types of centers are noted. Unless 
otherwise stated, all differences reported in the text are statistically significant. Statistically 
significant differences are identified by asterisks and the inclusion of p values in the report’s 
exhibits. “School-based” and “nonschool-based” refer to the location where grantees provide 
services for students. Moreover, although some centers serve only a single school level 
(elementary school, middle school or high school), many centers serve multiple school levels. 

All comparisons related to levels, therefore, analyze differences between centers serving a 
particular level (e.g., elementary) and those not serving that level (e.g., nonelementary). 
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2. Reading and Mathematics Activities in Centers Serving 

Elementary School Students 

Key Findings 

The following are key findings for centers serving elementary school students. Findings for 
centers serving middle and high school students appear in Appendix A. 

• Sufficient exposure to academics. To have an impact on academic outcomes, centers need 
to focus on academics, and students need to participate in academic activities at centers at 
a frequency that is sufficient to influence their learning (Dynarski et al., 2003; Fleming 
and Zhang, 2005). Grossman and colleagues (2002) found that homework help activities 
implemented well provided adult support to students and helped them to build academic 
skills. A recent meta-analysis found that after-school programs that provided one-on-one 
tutoring to students had positive effects on student academic achievement. The effect 
sizes ranged from 1 and 4/5 of a standard deviation, which are relatively large effects for 
educational programs (Lauer et al., 2006). More than 70 percent of centers serving 
elementary school students reported their students participated in reading activities on a 
weekly basis for at least two hours per week; 60 percent reported their students 
participated weekly in mathematics activities for at least two hours per week. Centers 
reported that a typical student spent 4 or more hours per week on reading activities (25 
percent of centers) and mathematics activities (19 percent of centers). Centers also 
reported that a typical student spent 4 or more hours per week working on homework (36 
percent of centers) and tutoring (12 percent of centers), both of which could include 
reading or mathematics activities. None of the observed activities involved one-on-one 
tutoring, but more than half of (58 percent) of observed reading activities involved some 
small-group (three to five students) tutoring activities (n = 23). 

• Focus on core academic content. Research in reading has identified core skills that are 
essential to literacy: phonemic awareness, phonics, vocabulary, fluency and 
comprehension (Snow, Burns and Griffin 1998). Likewise, research in mathematics has 
identified content strands that build students’ mathematical proficiency: number sense, 
operations, measurement, geometry, data analysis, statistics, probability and algebra 
(National Research Council, 2001, 2005). The data from surveys indicated that reading 
activities in centers serving elementary school students included all skills emphasized by 
the National Reading Panel (2000). Instruction in mathematics covered a narrower range 
of skills, with little emphasis on more complex problem solving. 

• Guidance from instructors. Instructors with backgrounds in helping struggling students in 
reading and mathematics are likely to be better prepared to offer high-quality instruction 
matched to student needs (Intercultural Center for Research in Education & National 
Institute on Out-of-School Time, 2005). Staff members’ active guidance of student 
learning is likely to be more effective than passive supervision of students (Brophy and 
Good, 1986; Good and Brophy, 2008; Rowan et al., 2002). Observers in case studies 
found that instructors in centers serving elementary school students used active teaching 
strategies in academic activities, communicating goals clearly to students in most 
activities (in 60 percent of reading activities and in 89 percent of mathematics activities). 




However, staff were less likely to use multiple teaching strategies in observed 
mathematics activities than in observed reading activities. The case studies were 
conducted in a small number of purposively sampled schools, so the data are not 
representative of all 21st CCLC sites. 

• Feedback to students and parents. Feedback to students helps develop their 

metacognitive skills and motivates improvement (Butler and Winne, 1995). Parent 
feedback supports caregiver involvement in student learning, a key component of 
academic success (Epstein, 2008; Sheldon, 2008). Most centers serving elementary 
school students reported providing frequent verbal feedback to students and parents, but 
written feedback was less common. 

Exposure to Academics After School 

Center activities need to provide sufficient exposure to academics if they are to increase the 
chances of improving student academic achievement (Dynarski et ah, 2003; Fleming and Zhang, 
2005). Reviews of academic time-on-task in schools (Wang, Haertel and Walberg, 1993) as well 
as evaluations of after-school programs (Baker and Witt, 1995; Huang, et ah, 2000) have found a 
relationship between sufficient exposure to academic content and a variety of positive academic 
outcomes. Lauer and colleagues’ (2006) meta-analysis found that reading programs with 
durations of 44 to 84 hours and mathematics programs with durations of 46 to 75 hours had the 
largest effect sizes. They also found that more exposure does not always result in higher 
academic achievement, making it difficult to say how much exposure to academics in an after- 
school program is ideal to improve academic achievement. 

According to the PPICS database, only 2 percent of all centers reported that they did not 
intend to offer activities that focused on reading, and 6 percent did not intend to offer activities 
that focused on mathematics. By contrast, although all the centers included in the survey sample 
intended to offer services in both reading and mathematics, of the surveyed centers serving 
elementary school students, 1 percent indicated that they did not offer reading activities and 4 
percent did not offer mathematics activities. Not all students in a center offering these activities 
are required to participate in both of these activities. 

Surveyed centers serving elementary school students reported that a typical student 
spent time weekly in academic support activities like homework and in reading or 
mathematics activities. Centers reported that a typical student participated in homework help 
(36 percent), reading activities (25 percent), mathematics activities (19 percent) and tutoring (12 
percent of centers) for more than 4 hours per week (see Exhibit 2-1; see also Exhibit E-1 in 
Appendix E). Additionally, 33 percent of centers reported that a typical student engaged in arts 
or recreation activities for more than 4 hours weekly. Because centers were open slightly less 
than 16 hours per week, on average, student participation of 4 hours per week in a particular 
activity amounted to about 25 percent of the available time. These data are different from and 
more detailed than those recorded in PPICS; one reason may be that PPICS data used for 
identifying the sample were recorded the year before the study team fielded the survey. 
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Exhibit 2-1 

Percentage of Centers Serving Elementary School Students That Reported Participation 
in Activities by a Typical Student, by Type of Activity and 
Amount of Time of Engagement per Week 




Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 5. 
n = 389 

Exhibit reads: Thirty-six percent of centers serving elementary school students reported that a typical student 
participated in homework activities for more than 4 hours per week. 



Reading Activities 

Reading Skills Emphasized 

Proficiency in reading requires a range of skills, including awareness of print features, 
phonemic awareness, phonics, vocabulary, fluency and comprehension (Snow et ah, 1998). 
Concentrating on only one of these reading skills is likely to be insufficient to improve reading 
achievement because evidence indicates that decoding and language comprehension skills make 
independent, but related, contributions to reading comprehension (Oakhill, Cain and Bryant, 
2003). 
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A majority of centers serving elementary school students reported that most or all 
reading activities’ included the skills emphasized by the National Reading Panel (2000). 

Center directors provided these data, which the study team asked them to obtain from someone 
familiar with the activities. Among centers serving elementary students, the largest percentage 
reported focusing on comprehension and vocabulary in all or most activities (74 percent and 69 
percent, respectively) (see Exhibit 2-2; see also Exhibit E-2 in Appendix E). 



Exhibit 2-2 

Percentage of Centers Serving Elementary School Students That Reported Reading 
Activities Emphases, by Reading Skill Emphasized and Inclusion Level 
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Note; Percentages may not equal 100 percent due to rounding. 

Source; National Survey, Item 22. 
n = 388 

Exhibit reads; Seventy-four percent of centers serving elementary students reported emphasizing comprehension 
in most or all instructional activities in reading. 



Reading Skills Emphasized in Observed Activities 

Site visit observations of centers that served elementary school students, though not 
nationally representative, were consistent with the pattern of the survey data. Of the 35 reading 
activity observations conducted, 86 percent stressed reading comprehension, and 71 percent 
emphasized vocabulary (see Exhibit 2-3). The observation data revealed a greater emphasis on 
fluency than did the survey data: 77 percent of observed reading activities focused on fluency, 
but 59 percent of centers reported that all or most reading activities emphasized fluency (see 
Exhibit 2-2). 



^ Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. 
Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment 
activities, but not homework assistance. 
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Exhibit 2-3 

Percentage of Observations of Reading Activities in 
Centers Serving Elementary School Students, by Reading Skili Emphasized 




Source: Site visit observations. 
n = 35 observations 

Exhibit reads: Eighty-six percent of observed reading activities in centers serving elementary school students 
focused on developing students’ comprehension skills. 



Use of Instructional Strategies for Teaching Vocabulary, Fluency and 
Comprehension 

The National Reading Panel (2000) summarized existing rigorous evidence on strategies for 

o 

the teaching of vocabulary, fluency and reading comprehension. To develop vocabulary skills, 
children need rich, explicit instruction (Beck, McKeown and Kucan, 2002), but children can also 
learn vocabulary incidentally, as part of authentic reading activity (National Reading Panel, 
2000). Both forms of instruction may be effective because children need extensive practice and 
multiple exposures to words in order to learn them (Coyne, et ah, 2004). Frequent opportunities 
to read aloud with guidance from staff, peers or parents have been found to have a significant 
and positive effect on word recognition, fluency and comprehension across a range of grade 
levels (Coyne, et ah, 2004). Finally, research suggests that strategies such as asking students to 
recall a text, answer and generate questions about it, make predictions about it and summarize it, 
improve comprehension (Biancarosa and Snow, 2006; National Reading Panel, 2000). Recent 
experimental research has also shown the advantages of selecting books at students’ reading 
level for them to read independently (Borman and Dowling, 2006). 



* This study focused on these components of reading in order to select survey items from the National Assessment of 
Educational Progress Teacher Background Survey related to reading instruction, and to design interview items that 
could measure the extent to which programs addressed the major processes involved in learning how to read, 
according to the findings of the National Reading Panel. 
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Although 99 percent of centers serving elementary school students reported using at 
least one strategy in reading, the most commonly reported reading activities were designed 
to foster more independent reading and greater comprehension. Roughly half of centers 
serving elementary school students indicated that in all or most reading activities, they asked 
students to explain or support statements about something they read (50 percent) or had students 
talk about or write answers to questions on something they had read (42 percent). More than a 
third of these centers also reported asking students to make predictions about a text they were 
about to read (38 percent), or to talk with peers about what they had read (36 percent) in all or 
most activities (see Exhibit 2-4; see also Exhibit E-3 in Appendix E). 

Exhibit 2-4 

Percentage of Centers Serving Elementary School Students That Reported Student 
Participation in Instructionai Strategies in Reading Activities, by Strategy and Inclusion 

Level 
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Source: National Survey, Item 21. 
n = 388 

Exhibit reads: Fifty percent of centers serving elementary school students reported having students explain or 
support their understanding of what they had read in most or all reading activities. 



2. Reading and Mathematics Enrichment in Centers Serving Elementary School Students 



18 






School-based centers were more likely than nonschool-based centers to report that their 
instructors asked students to make predictions about something they were reading, and talk about 
or write answers to questions related to something they had read. Observation data, while not 
nationally representative, were generally consistent with the survey data. Two instructional 
strategies were found in more than half of the observations conducted on site visits: having 
students help other students understand new words (80 percent) and asking students to make 
generalizations (63 percent) (see Exhibit 2-5). In contrast, as shown previously in Exhibit 2-4, 
only slightly more than a third of centers reported that most or all reading activities employed 
these strategies. At the same time, the proportions of observed activities in which students were 
asked to make predictions (49 percent) and read aloud (46 percent) were generally aligned with 
the survey data for these activities. 

Exhibit 2-5 

Percentage of Observations of Reading Activities 
in Centers Serving Elementary School Students, by Instructional Strategy Emphasized 
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Source: Site visit observations. 
n = 35 observations 

Exhibit reads: Eighty percent of observations of reading activities in centers serving elementary school students 
focused on students helping other students understand new words. 




2. Reading and Mathematics Enrichment in Centers Serving Elementary School Students 



19 






Mathematics Activities 

Mathematics Skills Emphasized 

The content of mathematics instruction is a significant predictor of gains in student 
achievement in mathematics (Gamoran, et ah, 1998; Porter, 2002; Rowan et ah, 2002). Syntheses 
of research have also pointed to the central importance of content in mathematics, drawing 
attention, for example, to the foundational role of numbers and procedures with numbers in 
mathematics (National Research Council, 2001, 2005). Instruction needs to emphasize more than 
just one strand of mathematics content, however; it is also important for students in the 
elementary and middle grades to learn about the foundations for algebra, which include a focus 
on fractions, measurement and geometry (National Research Council, 2001; National 
Mathematics Advisory Panel, 2008, pp. xvii, 18). The National Mathematics Advisory Panel 
recommends that the content should build over grade levels; failure to address any of the 
foundations, even in the early grades, may cause many students to fall behind (Roschelle, et ah, 
2009). In addition, large-scale studies of mathematics instruction have shown that the use of 
more rigorous content was a significant predictor of gains in student academic achievement 
(Gamoran et ah, 1998; Rowan et ah, 2002). 

Mathematics instruction focused on a relatively narrow range of content strands. 

Among centers serving elementary students, 71 percent reported emphasizing operations such as 
addition, subtraction, multiplication and division in all or most activities^ (see Exhibit 2-6; see 
also Exhibit E-4 in Appendix E). Eifty-nine percent reported that number sense (i.e., 
understanding of numerical relationships as expressed in ratios, proportions and percentages) 
was stressed in all or most activities. Operations with decimals or fractions, a key target of 
instruction according to the National Mathematics Advisory Panel, ranked third as an area of 
emphasis in center programming. Eewer activities targeted complex mathematics: 15 percent of 
centers emphasized algebraic concepts, and 8 percent focused on data analysis in all or most 
activities. 



^ Mathematics activities are defined as structured activities designed to build students’ mathematics skills. 
Mathematics activities may include scheduled time for independent problem solving and instructor-led enrichment 
activities, but not homework assistance. 
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Exhibit 2-6 

Percentage of Centers Serving Elementary School Students That Reported Emphases in 
Math Activities, by Type of Content Emphasized and Inclusion Level 
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Source: National Survey, Item 24. 
n = 387 

Exhibit reads: Seventy-one percent of centers serving elementary school students emphasized operations with whole 
numbers in most or all mathematics activities. 



School-based centers were more likely than nonschool-based centers to emphasize advanced 
topics in mathematics. In particular, school-based centers were significantly more likely than 
nonschool-based centers to have at least some emphasis on geometry (84 percent versus 70 
percent), algebra (75 percent versus 59 percent) and data analysis (70 percent versus 60 percent). 
However, from both types of centers, the reported focus on foundational topics such as number 
sense and operations was similar. 

Mathematics Content Emphasized in Observed Activities 

Observation data from the site visits, while not nationally representative, generally showed 
that centers primarily focused on teaching operations with whole numbers and number sense 
(see Exhibit 2-7). The emphasis on these basic skills may reflect a focus on building a strong 
foundation in basic math facts. In an interview at one of the centers visited, the mathematics 
instructor explained, “Because these kids are struggling, they need explicit systematic 
instruction. They have to get their foundation solid first.” In addition, a center’s ability to deliver 
instruction in higher-order mathematics may be limited by the knowledge of its tutors and staff. 
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In another center, one classroom teacher commented that tutors struggled with the level of 
content knowledge necessary to successfully assist older students with more complex 
mathematics problems, but did not experience the same challenges in literacy or language arts. 

Exhibit 2-7 

Percentage of Observations of Mathematics Activities in 
Centers Serving Elementary School Students That Emphasized Skills, by Type of Math 

Skill Emphasized 
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Source: Site visit observations, 
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Exhibit reads: Eighty-three percent of observed mathematics activities in centers serving elementary schools focused 
on developing students’ skills in performing operations with whole numbers. 
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Use of Instructional Strategies for Teaching Mathematics 

In mathematics, research on effective instructional strategies has focused on how best to 
develop the full range of skills students need to be mathematically proficient (National Research 
Council, 2001, 2005; National Mathematics Advisory Panel, 2008). One strand of research has 
examined the relationship between student academic achievement and the cognitive complexity 
of tasks instructors in regular classrooms assign to students. Cognitive complexity refers to the 
demands instructors place on students to engage in thinking about particular academic content. A 
low-complexity task in mathematics might involve asking students to memorize facts, definitions 
or formulas; in contrast, a high-complexity task might involve asking students to prove a 
theorem or solve a nonroutine problem that requires many steps (Porter, 2002). Although an 
emerging consensus views both procedural fluency and conceptual understanding as 
fundamental, interwoven aspects of mathematical proficiency (National Research Council, 

2001), strong experimental and correlational evidence indicates that when instructors require 
more complex tasks of students in a particular content area, student academic achievement gains 
are higher (Carpenter, et ah, 1989; Gamoran et ah, 1998). 
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Other evidence for effective strategies in mathematics comes from the National Mathematics 
Advisory Panel (2008), which was established to promote American students’ knowledge of 
mathematics and improve their performance in the subject. Having analyzed relevant research, 
the panel concluded that providing students the opportunity to explain the reasoning behind a 
problem solution helps to build proficiency in computation and translation of word problems and 
helps teachers to ensure that students have the conceptual framework necessary for 
understanding grade-level mathematics. The National Mathematics Advisory Panel (2008) also 
concluded that cooperative learning strategies, including peer-assisted learning strategies and 
heterogeneous grouping, have significant effects on student outcomes, particularly in teaching 
young children mathematical operations. Finally, other reviews of experimental research studies 
indicate that computers and graphing calculators can positively affect mathematics achievement 
(Dixon, et ah, 1998). 

Although 99 percent of centers serving elementary school students reported using at 
least one strategy, the most common strategies used in all or most mathematics activities 
were practicing mathematical facts (68 percent) and asking students to explain their 
reasoning in solving a problem (46 percent). Twenty-four percent of centers reported having 
students use computers to support mathematics instruction or to solve open-ended problems in 
mathematics (see Exhibit 2-8; see also Exhibit E-5 in Appendix E). 
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Exhibit 2-8 

Percentage of Centers Serving Elementary School Students That Reported Student 
Participation in Instructional Strategies in Math Activities, 
by Strategy and Inclusion Level 




Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 23. 
n = 388 

Exhibit reads: Sixty-eight percent of centers serving elementary school students reported asking students to 
practice basic mathematics facts in all or most instructional activities in mathematics. 



Mathematics Instructional Strategies in Observed Activities 

Observation data, though not nationally representative, showed a pattern similar to the survey 
data. Consistent with the mathematics strategies that centers reported on the survey, practicing 
basic mathematics facts was a strategy in 83 percent of the observed mathematics activities. 
Solving problems in which students had to select a method was assigned in 47 percent of 
observed activities, and students had to explain the reasoning behind their problem-solving 
strategy in 47 percent of observed activities. There were a few differences, however. For 
example, site visitors reported that students engaged in more complex tasks in mathematics than 
reported on the survey, but were asked to write mathematical equations less often than reported 
(see Exhibit 2-9). 
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Exhibit 2-9 

Percentage of Observations of Mathematics Activities in Centers 
Serving Elementary School Students That Emphasized Instructional Strategies, 

by Strategy Emphasized 
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Exhibit reads: Eighty-three percent of elementary school center mathematics activities observed involved students 
practicing basic mathematics facts. 
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Instructor Education and Experience 

Research on after-school programs suggests that when students receive guidance from 
instructors with education and experience relevant to their roles, students are more likely to 
benefit from program activities. A case study comparing after-school programs assessed to be 
“high quality” and “low quality” found that these programs differed on key program features like 
staff commitment, amount of staff training and staff background (Vandell, et al., 2004). A study 
of after-school programs in Massachusetts found that the programs that demonstrated higher 
outcomes relied on staff with high levels of educational attainment and often employed a number 
of instructors with teaching certificates (Intercultural Center for Research in Education & 
National Institute on Out-of-School Time, 2005). In addition to providing relevant expertise, 
certified instructors who also teach after-school participants during the regular school day may 
have a deeper knowledge of students’ needs (Little, 2006). 

Research on the need for certified instructors to offer high-quality instruction is by no means 
definitive, however. For example, a large-scale study of elementary schools found no 
relationship between student academic achievement gains and special certification for instructors 
for either reading or mathematics (Rowan et al. 2002). It is also important to note that the 
authorizing legislation for the 21st CCLC program does not require centers to use highly 
qualified teachers. 

Centers serving elementary school students reported that more than two-thirds of staff 
providing instruction in reading and mathematics had prior experience as certified 
classroom teachers or as instructional specialists in reading or mathematics. Fifty-two 
percent of the staff who provided instruction in reading and mathematics were at the time of the 
study or in the past regular classroom teachers, and 16 percent had been instructional specialists 
in reading or mathematics. Twenty-two percent of staff providing instruction in reading or 
mathematics were at the time of the study or had previously been classroom aides. 

During site visits, instructors in observed reading activities were more likely to hold a 
teaching certificate than instructors in mathematics activities. In addition, the percentage of 
certified teachers in reading was higher than reported in the survey. Currently certified teachers 
were present in 94 percent (n = 35) of observed reading activities (compared with 56 percent 
reported by the center surveys). Currently certified teachers were present in 50 percent (n= 18) 
of observed mathematics activities. The discrepancies may reflect the fact that the data from the 
observations were not nationally representative. 

Centers reported that nearly two-thirds of reading and mathematics instructors had a 
bachelor’s degree or higher (64 percent for reading and 63 percent for mathematics). 

Approximately one third of the centers indicated that their instructors had only a bachelor’s 
degree (32 percent for reading and 33 percent for mathematics) and a little more than a third of 
the centers said that their instructors had some graduate training (36 percent for reading and 37 
percent for mathematics). 
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Instructor Guidance in Observed Activities 

Research on active teaching has found that in classrooms where instructors actively 
supervise, instruct, support and provide feedback to students, students make greater learning 
gains in both reading and mathematics (Brophy and Good, 1986; Good and Brophy, 2008; 
Rowan, et ah, 2002). Active teaching is contrasted with teaching in which students primarily 
work independently or on tasks unrelated to academic activities (e.g., generating a list of 
behavior rules for the class). 



Instructors used active teaching strategies in 60 percent of all observed reading 
activities and in about 90 percent of all observed mathematics activities (see Exhibit 2-10). 

In 60 percent of reading activities, observers noted that instructors communicated the goals, 
purposes and expectations about activities to students. In more than half of all reading activities 
(53 percent), instructors used two or more strategies to teach a topic. In 89 percent of 
mathematics activities, observers rated instructors as providing guidance to students about the 
goals, purposes and expectations of activities. In 44 percent of observed mathematics activities, 
instructors challenged students to extend their current level of skill. In 22 percent of mathematics 
activities, instructors were observed to use more than one strategy to teach a skill. 



Exhibit 2-10 

Percentage of Observations in Reading and Mathematics Activities in Centers Serving 
Elementary School Students That Emphasized Use of Active Teaching Strategies, 
by Strategy Emphasized and Activity Type 
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Source: Site visit observations. 

n = 35 observations of reading activities; n = 18 observations of mathematics activities 

Exhibit reads: Instructors communicated the goals, purposes and expectations of activities to students in 

60 percent of the reading activities observed in centers serving elementary school students. 
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Feedback to Students and Parents on Student Progress 

Research on how people learn (National Research Council, 1999, 2005) emphasizes the need 
to provide feedback to learners about their progress. Feedback supports students’ developing 
skill in monitoring their own learning (Butler and Winne, 1995). Feedback can also prompt 
learners to correct misunderstandings (Bangert-Downs, et ah, 1991), motivate students’ interest 
in seeking help (Butler and Neuman, 1995) and increase students’ interest in improving their 
understanding or skill (Butler, 1987). Feedback to parents about students’ performance can also 
support improvement in learning and serve as an important vehicle for parent involvement 
(Epstein, 2008; Sheldon, 2008). Evidence suggests that delayed feedback — written comments on 
work as opposed to immediate verbal feedback — is better for long-term recall and retention 
(Butler and Winne, 1995). None of this research, however, was conducted specifically in after- 
school programs. 

Most centers serving elementary school students reported providing informal feedback 
to students; written or formal feedback was less common. Eor instance, 87 percent of school- 
based centers reported providing verbal feedback to students on work assigned by the school, as 
well as on their behavior. In contrast, about a quarter of school-based centers gave students 
written feedback on work assigned by the school (22 percent) and on work assigned in the center 
(26 percent). More than two-thirds of nonschool -based centers reported offering some kind of 
reward or recognition for program accomplishments, more than did school-based centers (see 
Exhibit 2-11; see also Exhibit E-6 in Appendix E). 



Exhibit 2-11 

Percentage of Centers Serving Elementary School Students That Provided Feedback 
About Their Progress, by Type of Feedback and Center Type 
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Center Type 
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based 


School- 

based 


Verbal feedback from center staff 


On work assigned by the school 


86% 


87% 


On work assigned at the center 


74% 


72% 


On student behavior 


87% 


87% 


Written feedback from center staff 


On work assigned by the school 


26% 


22% 


On work assigned at the center 


34% 


26% 


On student behavior 


42% 


48% 


“Points” or rewards for program accomplishments 


65%* 


48%* 


Certificates or awards for accomplishments 


72% 


62% 



Source; National Survey, Item 18. 



n = 393 
*p < 0.05 

Exhibit reads; Eighty-six percent of nonschool-based centers serving elementary school students provided verbal 
feedback to students on work assigned by the students’ schools. 



2. Reading and Mathematics Enrichment in Centers Serving Eiementary Schooi Students 



28 




A majority of centers serving elementary school students reported providing frequent, 
informal feedback about student progress to parents. Sixty-two percent of centers serving 
elementary students reported talking with parents nearly every week when the parents picked up 
or dropped off their child. Four percent sent home written reports of students’ progress in the 
after-school program nearly every week, but 53 percent never did so or did so rarely. One 
percent of centers serving elementary school students indicated that they held parent-center staff 
conferences nearly every week, and 56 percent never or rarely held conferences. Nonschool- 
based centers were more likely to hold parent-center staff conferences than school-based centers, 
but less likely to contact parents by telephone or e-mail. These differences were statistically 
significant. 

Summary 

Overall, the reading and mathematics activities implemented in 21st CCLC programs that 
serve elementary students reflected a focus on academics. Most students were exposed to a 
variety of activities in reading and mathematics. Additionally, centers reported offering feedback 
to students and parents on a regular basis about their progress. The feedback was largely 
informal, however, and limited primarily to verbal exchanges about progress on work assigned 
by the school. 

Reading activities covered a broader range of skills and used a wider variety of strategies 
than did mathematics activities. For example, instruction covered a broader range of reading 
skills, and centers were likely to use a range of strategies for teaching reading skills. In contrast, 
mathematics instruction focused primarily on two content strands and engaged students in tasks 
with low levels of cognitive complexity. The emphasis on operations and number sense in 
mathematics activities may have resulted from a need to build a strong foundation of basic 
mathematics skills among students. These differences in the quality of reading and mathematics 
activities cannot be explained using data from the surveys about staff qualifications, because 
instructors in reading and mathematics had similar levels of formal education. 
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3. Student Targeting and Attendance 

In general, 21st Century Community Learning Centers operate before and after school as 
well as during the summer. According to a recent report (Naftzgeret, et ah, 2006), the operating 
hours for centers (typically open Monday through Thursday during the school year) averaged 
14.2 hours per week during the school year and 23.5 hours per week during the summer. On 
average, centers were open for 150 days during the school year. 

Key Findings 

• Setting priorities and policies regarding services and participation. Centers’ policies are 
important organizational supports, particularly for setting expectations regarding student 
attendance (Crollick, Zhang and Fleming, 2005). More than half of all centers had 
policies that required students to attend, but mandatory attendance policies were not 
found to relate to greater attendance. About two-thirds of centers had tracking 
systems capable of identifying how many days individual students participated in the 
program (i.e., could generate attendance reports easily, within 1-2 hours’ time). 

• Regular attendance. Regular attendance by students who are most in need of after-school 
programming is necessary for them to benefit from the instruction (Reisner, et ah, 2004). 
Although all centers served students who attended Title I schools, the centers did not 
focus solely on serving low-performing students. Most allowed any interested students to 
participate. School-based centers were more likely to give priority to low-performing 
students. 

• Attendance rates and grade level. Of the centers that could generate daily attendance 
reports easily (within 1-2 hours’ time) (75 of 122 indicated that they could generate 
attendance reports easily), centers serving elementary school students reported that 48 
percent of students attended 60 days or more in 2005-06. Centers serving middle grades 
reported 36 percent of students attended this often, and centers serving high school 
students reported 30 percent of their students met this standard. These findings are not 
generalizable to all centers; it is possible that centers that are capable of reporting 
attendance data may be more likely to have higher attendance. 

• Student engagement in academic activities. Beyond attendance, being on task and 
focusing on academics when present support students’ motivation to learn and appreciate 
the importance of developing proficiency in reading and mathematics (Chaput, Little and 
Weiss, 2004; Mahoney, Parente and Lord, 2007). Students were more engaged in 
observed reading activities during site visits than in observed mathematics activities. 
However, these data are not nationally representative. 



The National Survey Item 10 asked center directors: “What policies regarding attendance does the center have for 
students?” Possible answers were: “Attendance is not required; students can drop in and out”; “We require students 
to attend at least 1 day per week”; “We require students to attend 2-3 days per week”; “We require students to 
attend every day we offer services”; and “Other.” 

* * Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations 
of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a 
separate question about consequences of nonattendance. 




Priorities Regarding Targeting 

Historically, economically advantaged students have had greater access to after-school 
programs than students from low-income families (Simpkins, et ah, 2005). The legislation that 
authorizes the 21st CCLC program requires grantees to primarily target students who attend Title 
I schools in which 40 percent or more of the students are from low-income families and the 
families of such students. At the same time, the program gives local grantees some discretion in 
how their programming targets students who attend these schools. 

In general, centers reported that they sought to serve a variety of students and used a mix of 
criteria to target students for participation. Overall, 85 percent of centers reported that they 
sought to serve students identified by the school as needing special assistance in reading and/or 
mathematics, and 76 percent sought to serve students scoring “below proficient” on state tests. 
Sixty-eight percent of centers sought to serve students who were eligible for free or reduced- 
price lunches. 

When centers were asked to rank their top three priority groups, the most commonly reported 
groups were students who scored below proficient on state assessments (44 percent of centers) 
and students whom the school identified as needing special assistance in reading and 
mathematics (21 percent of centers). By contrast, a relatively small proportion of centers (2 
percent of school-based centers and 3 percent of nonschool-based centers) reported that students 
who received free or reduced-price lunch were among the top three priority groups. 

Top-priority groups differed in certain respects for school-based and nonschool-based centers 
(see Exhibit 3-1; see also Exhibit E-7 in Appendix E). Both types of centers targeted students 
scoring below proficient on state tests, but nonschool-based centers were more likely than 
school-based centers to target this group (79 percent versus 46 percent). Nonschool-based 
centers were also more likely than school-based centers to report their top priorities were 
students whom their teacher or counselor identified as in academic need (88 percent versus 55 
percent). 
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Exhibit 3-1 

Percentage of Centers That Reported Top-Priority Target Groups, 
by Target Groups and Center Type 
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Source: Focused Telephone Survey, Item 1. 
n= 102 
*p < 0.05 

Exhibit reads: A higher percentage of centers in school-based settings (75 percent) reported they had open 
enrollment for all interested students than nonschool-based centers (60 percent). 



Policies Regarding Attendance 

Data on general attendance policies were collected for all centers, but data on attendance 
were collected using the focused survey from a random sample of centers sampled for the 
national survey. The study team asked centers to report attendance only if they had adopted 
methods for tracking attendance and only if staff kept daily records of individual students’ 
attendance. Sixty-one percent of the centers tracked attendance in a way that allowed them to 
report easily (within 1-2 hours’ time) how many days an individual student attended in the past 
year. 

More than half of all centers (56 percent) in the full sample of centers reported having 
policies that require students to attend 2 days per week or more to remain enrolled in the 
program. Thirty-eight percent of centers required daily attendance, and 26 percent of centers 
reported that attendance was on a drop-in basis. Centers serving elementary school students were 
more likely to have policies that required attendance than did those serving older students (see 



Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations 
of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a 
separate question about consequences of nonattendance. 
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Exhibit 3-2; see also Exhibit E-8 in Appendix E). Eorty-one percent of centers serving 
elementary school students required students to attend the program every day services were 
offered, compared with 3 1 percent of centers serving students in middle school and 22 percent of 
centers serving high school students. These data are for the sample that completed the national 
survey. 



Exhibit 3-2 

Percentage of Centers That Reported Attendance Requirements, 
by Grade Level and Attendance Requirement 
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Note; Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 10. 
n = 499 
*p < 0.05 

Exhibit reads: Forty-one percent of centers serving elementary school students reported that they required 
the students to attend every day the center-offered services. 
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More than half the centers reported requiring participation for all students in specific 
academic activities when students were present, including homework help (73 percent), 
reading activities (60 percent), mathematics activities (58 percent) and tutoring (14 percent) (see 
Exhibit 3-3; see also Exhibit E-9 in Appendix E). Programs serving high school students were 
less likely than those not serving this age group to require participation in these activities. A 
higher proportion of school-based centers reported that they required participation in reading 
activities (10 percent, compared with 2 percent of nonschool-based centers) for all students 
attending the center. 



Exhibit 3-3 

Percentage of Centers That Reported That They Required Participation of All Students in 
Specific Activities When Students Were Present, by Type of Activity 




Source: Focused Telephone Survey, Item 9. 
n= 114 

Exhibit reads: Seventy-three percent of centers reported that homework time was mandatory. 



Center directors at case study sites noted that they felt it important to consider carefully what 
would engage students and encourage them to keep coming to the program. Eor example, in a 
Elorida center, the director offered engaging recreational activities as a way to entice students to 
attend the academic components of the program, which were voluntary. “The main issue is if you 
only offer academics, then your attendance is affected.. . . We’ve offered a professional football 
player doing flag football games, and that generated so much interest among kids. . . . They’re 
dying to play and consequently are attending after school.” 

Regular Attendance 

Research on after-school programs generally has found a significant association between 
regular program attendance and positive student outcomes. Eor example, participation in after- 
school activities sponsored by The After School Corporation in New York City was linked to 
higher grades and increased school attendance, especially for students who participated regularly 
in the corporation’s programs over 2 consecutive years (Reisner, et ah, 2004). Researchers 
studying attendance in before- and after- school programs have argued that it is important for 
evaluation studies to distinguish among intensity, duration and breadth of student participation 
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(Chaput, et al., 2004). Intensity refers to how often students participate in a program during a 
given time period. Duration refers to the length of time (in days, months or years) students 
participate in programs. Other studies have found a relationship between regular attendance in 
after-school programs and educational attainment in high school (Fredricks and Eccles 2006), in- 
class attendance (Grossman et al., 2002; Huang et ah, 2000) and reading and mathematics 
achievement (Lauer et al., 2006). 

Although most research tends to focus on just one aspect of participation, each type of 
measure has been associated with improved outcomes for youth. For example, in their evaluation 
study of two after-school programs. Baker and Witt (1995) found a relationship between 
intensity of participation and a variety of academic outcomes (see also Huang, et al., 2000). 
Duration appears to be particularly significant for arts-related programming, where interactions 
may not be intense but take place over many years (Catterall, 1998). A study examining the 
relationship between attendance and academic achievement of students participating in Los 
Angeles’ BEST after-school program found that elementary school students attending program 
activities for 50 or more days throughout the school year demonstrated positive academic 
achievement growth (Huang, et al., 2008). Other researchers have studied how attendance in 
after- school programs, in combination with other home-based and community activities, is 
linked to positive behavioral and academic outcomes (Vandell, Reisner and Pierce, 2007). More 
attendance is not always better, however, as Lauer and colleagues (2006) reported in their 
synthesis: Students who received more than 210 hours of instruction did not show gains in 
reading. 

As stated earlier, 21st Century Community Learning Centers operate before and after school 
as well as during the summer. According to Naftzger and colleagues (2006), centers were open 
an average of 14.2 hours per week during the school year and 23.5 hours per week during the 
summer. On average, centers were open for 150 days during the school year. 

Regular attendance has been a concern within the 21st CCLC program, especially since the 
publication of the first national evaluation (Dynarski, et al., 2003). That study found that many 
programs had lower-than-expected attendance: 55 percent of students in the elementary school 
treatment group of this evaluation attended 5 1 or more days per year, and 20 percent of middle 
school students in the treatment group attended that often. 

Although this study sought to better understand the nature of attendance in centers, the data 
collected from this study may be of limited utility in addressing concerns about attendance raised 
within the 21st CCLC program, since it was not collected in the same way as the national 
evaluation and did not distinguish among different types of programs (summer, before- or after- 
school programs). 

Of the centers that reported on attendance in the current study (approximately 61 
percent of centers surveyed), 44 percent of students attended 60 days or more in the last 
year. Centers serving elementary school students reported higher attendance than did centers 
serving middle and high school students (see Exhibit 3-4; see also Exhibit E-10 in Appendix E). 
Centers serving elementary school students reported that 48 percent of students attended 60 days 
or more in the last year, which amounts to approximately 2 days a week. Centers serving middle 
grades reported 36 percent of students attended this often (60 days or more), and centers serving 



3. Student Targeting and Attendance 



36 




high school students reported 30 percent of their students met this standard. In the previous 

evaluation, the average elementary student attended programming 2 to 3 days a week or 56 days 

per year during the year of the study (Dynarski et ah, 2004), and the typical middle school 

student attended 1 day a week, or 32 days a year (Dynarski et ah, 2003). The results of this study 

cannot be compared directly with the results of the previous evaluation because the two studies 

1 ^ 

did not measure attendance using the same method. 



Exhibit 3-4 

Percentage of Centers That Reported the Number of Days of Attendance in 2005-06, 

by Grade Level and Days of Attendance 




Note; Percentages may not equal 100 percent due to rounding. 

Source: Focused Telephone Survey, Item 8. 
n = 15 

Exhibit reads: Centers serving elementary school students reported that 17 percent of students attended 120 days or 
more. 

The proportion of students attending 60 days or more was higher in school-based 
centers than in nonschool-based centers (52 percent versus 44 percent). A third of students 
in school-based centers serving elementary school students attended fewer than 30 days, 
compared with 14 percent in nonschool-based centers serving elementary school students. About 
half of students in school-based centers serving high school students attended fewer than 30 
days, compared with 89 percent in nonschool-based centers serving high school students. 



Data collection methods varied between this study and the previous study. The previous national evaluation 
collected program attendance records of individual students across 2 years, while the current study relied on survey 
reports of attendance during 1 year by program directors. Hence, the attendance rates in the previous evaluation are 
likely to be more accurate than the self-reported attendance rates reported in this study. 
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Overall attendance rates were similar for centers that did and did not require some 
level of attendance to remain enrolled in the program, hut attendance policies were related 
to attendance in centers serving secondary-level students. In centers that required some level 
of attendance, 66 percent of students attended for more than 30 days; 62 percent attended for the 
same duration in centers that did not require attendance. There was no relation between any kind 
of attendance policy and actual attendance for centers overall or for centers serving elementary 
school students. For the middle grades, centers that required attendance every day had higher 
attendance than did centers with other attendance policies. In high school, centers that required 
attendance 2 to 3 days per week had higher attendance. 

Staff members address absences from the programs primarily by talking with parents 
and students (see Exhibit 3-5; see also Exhibit E-1 1 in Appendix E). Eifty-eight percent of staff 
members reported they notified parents or caregivers if the student did not meet the attendance 
requirements of the center. Thirty-eight percent said they spoke to students after several 
absences. Eew centers punished students for absences by restricting privileges (6 percent) or by 
restricting participation in particular activities (10 percent). 

Exhibit 3-5 

Percentage of Centers That Reported Consequences for Student Absences, 

by Consequence Type 
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Source: National Survey, Item 11. 
n = 503 

Exhibit reads: Fifty-eight percent of centers reported notifying parents when a student is absent from the program. 
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student Engagement in Observed Activities 

Beyond attendance, learning requires student engagement in after-school activities (Chaput et 
al., 2004; Mahoney et al., 2007). Cognitive engagement in learning tasks is closely related to 
self-regulation in learning: by engaging with classroom activities, students are better able to track 
new information and assimilate feedback from instructors (Corno and Mandinach, 1983). 
Engagement is also related to students’ beliefs about the value of effort in learning (Mahoney et 
al., 2007; Pintrich and Schrauben, 1992). Motivation and engagement are important to 
developing students’ disposition to see mathematics as valuable and worthwhile (National 
Research Council, 2001) and in developing students’ reading skills (National Reading Panel, 
2000 ). 

Observers in the case studies rated students as “on task” in most activities in reading, 

but not in mathematics. In 80 percent of reading activities, observers judged students to be on 
task and engaged in those activities, compared with 42 percent of mathematics activities. 

Summary 

Centers reported seeking to serve low-income students and students who scored low on state 
assessments or whom instructors judged to need special assistance in reading and mathematics. 

Centers’ reports on student attendance revealed a familiar pattern. On average, elementary 
schools attended programming more than middle and high school students did. With respect to 
regular attendance, this evaluation found that, on average, elementary students attended 
programming more than middle and high school students did. These findings replicated a general 
trend shown in other studies (Borman and Dowling, 2006; Grossman et al., 2002; Reisner et al., 
2004; Simpkins et al., 2005): Elementary school students attended after-school activities for 
more days than did middle and high school students. These data underscore the challenge centers 
face in encouraging and sustaining regular student attendance. 

Engagement was generally high in reading activities but lower in mathematics activities. 
These data help underline the differences between these two subjects. Observers in the case 
studies rated students as “on task” in most activities in reading, but not in mathematics. Eower 
engagement in mathematics activities may be related to the narrower range of mathematics skills 
targeted by these activities. 
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4. Organizational Supports for Instructional Quality 

Key Findings 

• Center staffing. Research on staff compensation suggests that adequate pay and benefits 
can help attract and retain qualified staff to support high-quality after-school 
programming (Wechsler et ah, 2001). Additionally, it is important for after-school 
programs to provide sufficient staff development to enhance staff skills and maximize 
their expertise relevant to their roles (Crollick et ah, 2005). Centers relied primarily on 
part-time staff, who were unlikely to receive benefits as a result of their work in the 
center. This is not surprising, since centers offered programming 16 hours per week, on 
average. It is also possible that some of these part-time staff were employed full-time as 
teachers or paraprofessionals at the time of the study. The majority of centers reported 
providing paid professional development opportunities to full-time and part-time staff. 
Across all centers, 29 percent of the staff had worked at the center for less than 1 year. 
About half of centers (48 percent) reported that the primary reason for staff turnover was 
graduation from school or completion of a program of study. 

• Alignment of activities with student needs. After-school programs provide an opportunity 
to extend students’ learning beyond the school day, but the effectiveness of that help 
depends, in part, on the quality of the relationships between adults and students 
(Birmingham et ah, 2005; Carruthers and Busser, 2000; Grossman et ah, 2002; Huang et 
ah, 2000; Intercultural Center for Research in Education & National Institute on Out-of- 
School Time, 2005; Pierce, Hamm and Vandell, 1999). Centers reported using state 
assessment data to assess student growth and to identify different strategies for matching 
activities intended to meet student needs. About one-quarter of the centers coordinated 
activities with supplemental educational services (SES) providers; 9 percent reported 
coordinating schedules and academic activities with providers. More than two-thirds of 
centers reported having access to whole-school achievement data for reading and 
mathematics. 

• Coordination with the school day. High-quality instruction is aligned with student needs; 
however, to understand student needs, after- school programs need to be well coordinated 
with school-day activities and use data from student assessments to adjust program 
offerings (de Kanter, et ah, 2003; Noam, 2003). Respondents from nonschool-based 
centers were more likely than school-based centers to report challenges aligning activities 
with the school day because they lacked information about students’ needs and because 
school staff were not responsive to the centers’ requests. 

Center Staffing 

Recruiting and Retaining High-Quality Staff 

An important role for center directors in 21st Century Community Eeaming Centers 
programs is recruiting and retaining high-quality staff with expertise matched to their roles 
(Crollick et ah, 2005). Past research has focused primarily on the role of competitive salaries in 
retaining staff (Halpern, Barker and Mollard, 2000). This section focuses on both staff pay and 
benefits at centers and on center directors’ perceptions about the relationship between staff pay 
and benefits and staff turnover in centers. 




About half of centers paid staff with current teaching certificates $15 to $25 per hour 
(54 percent) and staff without current teaching certificates $10 to $15 per hour (52 percent) 

(see Exhibit 4-1; see also Exhibit E-12 in Appendix E). Currently certified teachers who worked 
in nonschool-based sites were paid less than staff without teaching certificates working in 
school-based settings, but pay for staff without teaching certificates was comparable across 
center types. Data from the Bureau of Eabor Statistics (Occupational Online Network, 2008) 
indicate that the median wage for paraprofessionals working in schools in 2008 was 
approximately $1 1.45 per hour (based on an annual salary of $22,500). Rates are not likely to be 
comparable across different regions of the country; however, no objective data are readily 
available about how competitive wages are for center staff. 



Exhibit 4-1 

Percentage of Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, 

by Type of Staff and Rates 
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Exhibit reads: One percent of all centers paid staff who are certified teachers less than $10 per hour. 
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Seventy-six percent of staff in all centers reported working fewer than 20 hours per 
week at the center (see Exhibit 4-2; see also Exhibit E-13 in Appendix E). Ten percent of staff 
members offering academic instruction worked in full-time positions. Another 14 percent 
worked half time or more. Nonschool-based centers were more likely to employ staff in half- 
time (21 percent) or full-time (34 percent) positions, compared with school-based centers (11 
percent half-time and 8 percent full-time). Centers serving high school students were more likely 
to have staff members who worked full-time than were centers not serving this age group. Center 
staff could be employed in other jobs, such as working as teachers during the school day. 



Exhibit 4-2 

Percentage of Staff Who Led Instructional Activities at Centers, 
by Hours Worked per Week 




Note: Percentages may not equal 100 percent due to rounding. 

Source: Focused Telephone Survey, Items 26, 32 and 38. 
n = 349 

Exhibit reads: Ten percent of staff worked full-time at the center (more than 35 hours 
per week). 
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About half of centers reported offering some type of benefits to either full- or part-time 
staff members. The most commonly offered benefits related to health insurance (offered to full- 
time staff in 43 percent of centers and to part-time staff in 8 percent of centers) and retirement 
plans (offered to full-time staff in 41 percent of centers and part-time staff in 14 percent of 
centers) (see Exhibit 4-3; see also Exhibit E-14 in Appendix E). About half of centers reported 
that they did not offer health insurance (49 percent) or paid time off for vacation and sick leave 
(49 percent) to any staff members. Nonschool-based centers were more likely than school-based 
centers to provide their full-time staff with health insurance (66 percent versus 40 percent), a 
retirement savings plan (59 percent versus 39 percent) and paid time off for vacation and sick 
leave (72 percent versus 37 percent). Centers serving high school students were more likely to 
provide their full-time staff with health insurance and paid time off than centers not serving this 
age group, but less likely to provide this benefit to their part-time staff than centers not serving 
high school students. Part-time staff at centers serving high school students were also less likely 
than staff at centers not serving this age group to receive tuition reimbursement as a benefit. 
Einally, centers serving middle school students were more likely than centers not serving this 
group to provide health insurance to their part-time staff. 



Exhibit 4-3 

Percentage of Centers That Reported Offering Benefits to Full- and Part-Time Staff Who 
Led Instructional Activities, by Benefit Offered and Staff Status 
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Source: National Survey, Item 28. 
n = 505 

Exhibit reads: Forty-three percent of centers offered health insurance to full-time staff. 
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Over one- quarter of staff members had been in their positions for less than 1 year. 

Across all centers, 29 percent of the staff had worked at the center for less than 1 year.’"^ 

The most commonly reported reasons for staff turnover were graduation from school 
or completion of a program, but staff also left for lack of benefits and because centers were 
unable to offer a full-time position. More nonschool-based centers than school-based centers 
said that staff left the program because of the lack of benefits (44 percent versus 25 percent) and 
because of low pay (18 percent versus 8 percent) (see Exhibit 4-4; see also Exhibit E-15 in 
Appendix E). 



Exhibit 4-4 

Percentage of Centers That Reported Reasons for Staff Departures, 
by Reason for Departure and Center Type 
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Source; National Survey, Item 30b. 
n = 466 
*p < 0.05 

Exhibit reads: Forty-nine percent of school-based centers reported that the primary reason why staff left was 
because they had completed a program (e.g., AmeriCorps) or graduated from school compared with 42 percent of 
nonschool-based centers. 



Professional Development 

Ensuring staff quality in schools is widely believed to require ongoing professional 
development in evidence-based practice (National Reading Panel, 2000; National Research 
Council, 2001). Similarly, researchers of after-school programs have argued that professional 
development is necessary for staff to learn principles of youth development (Huebner, Walker 



Since the National Survey Item 29 asked only about “all staff,” the tenure for certain types of staff may vary. 
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and McFarland, 2003), develop skills for building positive relationships with youth (Carruthers 
and Busser, 2000) and acquire strategies for supporting youth skill building (Carruthers and 
Busser, 2000). For programs seeking to enhance students’ academic achievement, professional 
development activities need to be aligned with student needs identified by school-day staff and 
administrators (Huang, 2001). An analysis of The After School Corporation’s most promising 
programs with respect to youth outcomes indicated that those programs all had a leader who 
provided ongoing opportunities for professional development to staff members (Birmingham et 
ah, 2005). 



The majority of centers reported providing paid professional development 
opportunities to full-time and part-time staff (see Exhibit 4-5; see also Exhibit E-16 in 
Appendix E). Professional development often included conference attendance paid for by the 
center (53 percent of centers for full-time staff and 39 percent of centers for part-time staff). 
Centers were more likely to pay for training or professional development to full-time staff (62 
percent) than to part-time staff (55 percent). At the same time, for both full-time and part-time 
staff, school-based centers were less likely than were nonschool-based centers to pay for 
attendance at conferences. Centers serving students in the middle grades were less likely to pay 
for attendance at conferences to their part-time staff than were centers not serving this age group. 



Exhibit 4-5 

Percentage of Centers That Reported That They Offered Paid Professional Development 

Opportunities for Full- and Part-Time Staff 
Who Led Instructional Activities, by Staff Status and Type of Development 
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Source; National Survey, Item 28. 
n = 505 

Exhibit reads: Sixty-two percent of centers reported that they offered paid professional development to full- 
time staff. 



Site visit data suggest that professional development offered hy centers may have 
focused on nonacademic areas. Eour of 12 case study centers provided just 1 to 2 days of 
training or orientation to staff members, and that training focused primarily on logistics and 
program implementation. At first glance, these findings appear to contradict the survey findings, 
but the survey did not include questions about the content of the professional development. 
Additionally, the case study data are not nationally representative. 
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Among the case study sites, few centers reported that they offered professional development 
to staff, beyond that which was available to regular school-day staff members. Center directors in 
the case studies said they relied on the professional development opportunities their instructional 
staff received through the district, and they thought additional professional development specific 
to the center’s program might be a burden for staff. For example, one director noted that most of 
the center’s instructional staff were “already certified instructors who have to go to so many 
workshops that we don’t feel the need to give them more professional development.” During site 
visits, directors reported that rather than participating in formal professional development 
workshops or conferences, staff were encouraged to share their ideas for innovative 
programming in weekly staff meetings. 

Alignment to Student Needs 

Coordination With the School Day 

A central challenge for centers is to coordinate their activities with the school-day 
instructional program in ways that are aligned with student needs and with the school-day 
curriculum (de Kanter et ah, 2003; Noam, 2003). Studies of after-school programs show that 
when programs align their activities to complement school-day activities, they can contribute to 
student success (Little, 2006). At the same time, studies also indicate that such coordination can 
be difficult, especially when programs are not based in schools, because the programs may have 
limited access to the curricular materials used in the school day and to school-day staff who 
could inform them about particular students’ progress and challenges. 

More than two-thirds of the centers (71 percent) had access to whole-school state 
assessment data in reading, and 69 percent had access to this information for mathematics. 

Of those centers that received these data, 83 percent also had access to individual students’ level 
of proficiency on state assessments. 

Significantly more school-based centers than nonschool-based centers had access to 
whole-school data. More school-based centers reported having access to state assessment data at 
the school level on reading (73 percent) and mathematics (71 percent) than did nonschool-based 
centers in those two subject areas (53 percent and 52 percent, respectively) (see Exhibit 4-6; see 
also Exhibit E-17 in Appendix E). Centers did not differ by type with respect to their access to 
individual-level data, however. 
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Exhibit 4-6 

Percentage of Centers That Reported That State Assessments Data Was Available F 
for Use in Tracking Student Outcomes, by Subject Tested and Center Type 




Source: National Survey, Item 13. 
n = 467 
*p < 0.05 

Exhibit reads: Seventy-three percent of school-based centers reported that they had access to test results from No 
Child Left Behind (NCLB) state reading assessments for the whole school. 



Use of Student Assessment Data 

Instructors should be able to assess the needs of individual students and tailor instruction to 
meet specific needs (National Reading Panel, 2000). Likewise, in mathematics, emphasis should 
be on assessing and engaging students’ prior knowledge (National Researeh Couneil, 1999, 
2005). In terms of providing quality programming, a key use of assessment data, then, is to 
adjust instructional practice and design new offerings that better meet identified student needs. 

Center directors reported using several sources of information to assess student 
academic growth, make program adjustments or evaluate program success. Almost half of 
the centers reported using results from tests administered at students’ sehools to assess students’ 
academic growth in the 21st CCLC program. More centers reported using informal staff 
observations (28 percent of eenters reported using them almost daily) and verbal reports from 
teaehers (13 percent reported using them almost daily) to assess student progress with greater 
frequency than more formal assessments. Only 6 pereent of centers reported that they used 
results from tests administered by center staff or student self-assessments (17 pereent) at least 
weekly to assess students or evaluate program success (see Exhibit 4-7; see also Exhibit E-18 in 
Appendix E). 
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Exhibit 4-7 

Percentage of Centers That Reported Student Assessment Sources, 
by Type and Frequency of Assessment 




Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 15. 
n = 502 

Exhibit reads: Twenty-eight percent of centers reported using informal staff observations on an almost daily basis. 



More than half of the centers reported using assessments of students’ academic growth 
for at least one of the following purposes: to identify new offerings (59 percent) or improve 
practices in existing program offerings (69 percent), to assign students to particular staff 
members (54 percent) and to evaluate the success of their programs (66 percent)Two-thirds 

of centers reported using assessment information to improve practices in existing program 
offerings (see Exhibit 4-8; see also Exhibit E-19 in Appendix E). 

Nonschool-based centers were significantly more likely than school-based centers (67 
percent versus 52 percent) to report using formative assessment data collected by the center to 
assign students to particular staff members who could meet their specific academic needs. Eor 
example, at one site visited, staff used data on students’ mathematics assessments administered 
by the center to assign them to mathematics tutors who were also certified instructors. Centers 
serving elementary school students were significantly less likely than were centers not serving 
this age group to use center-administered assessment data to identify new program offerings (56 
percent of centers serving elementary school students versus 60 percent of centers serving 
middle school students and 64 percent of centers serving high school students). 
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Exhibit 4-8 

Percentage of Centers That Reported Uses of Student Assessment Data, by Type of Use 




Source; National Survey, Item 16. 
n = 516 

Exhibit reads: Sixty-nine percent of school-based centers reported using students’ assessments of academic growth 
to improve existing program offerings. 

Case Study data illustrate ways that programs used data to target instruction. For example, 
one Florida grantee used extensive student-level data to track student progress and growth, 
including results from the two benchmark assessments administered by the district. Staff in this 
center looked at the subscale scores to identify areas of weakness in students’ skills and 
determine which skills to target during the after-school program. Similarly, a North Carolina 
grantee used the pre- and post-test data associated with the reading and mathematics intervention 
curricula provided by the center to monitor student progress from week to week. The center 
leadership worked with staff members to identify the trends in these data and adjust instruction 
to meet students’ needs. 

Beyond the use of individual assessment to target specific instruction to students, the case 
study data support the survey findings that staff in some centers used data for program- level 
improvement. For example, one Maryland grantee employed an onsite evaluator to monitor 
program activities. That program evaluator used a narrative reporting form to document weekly 
program visits, as well as surveys of parents, students and classroom instructors. The evaluator 
then used the data collected by these means to “zero in on the [progress related to] homework 
and academic performance and to assess how students are getting along” in order to “analyze it 
by school and communicate it back to the coordinators at each site.” On the basis of this 
feedback, the coordinators adjusted programming activities, targeting students in need. 
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Use of data by some centers in the case studies was more limited. For instance, a site in 
South Dakota made little use of student assessment data. According to the center director, staff at 
this nonschool-based site did not use data in their decision-making processes. All data collected 
were used for public relations or for reporting to grantee agencies. The outreach director reported 
that he did not find the data collected to be useful: “I found next to no useful information from 
that evaluation. . . . For me, the purpose of evaluation is to make the program better, and if I’m not 
going to learn how to make the program better, I don’t need it.” 

Despite the achievement data available to many centers, about a third of centers 
reported lacking information about students’ academic needs, a lack of responsiveness by 
school staff to center requests and a lack of information about the school’s curriculum as 
major challenges in implementing high-quality programming. Nonschool-based centers were 
more likely to report these challenges than were school-based centers (see Exhibit 4-9; see also 
Exhibit E-20 in Appendix E). A significantly higher percentage of nonschool-based center 
directors (36 percent) reported they had difficulty obtaining information about student needs 
from schools than did directors of school-based centers (20 percent). Similarly, 39 percent of 
nonschool-based centers reported a lack of responsiveness by school staff to their requests, 
compared with 29 percent of school-based centers. Directors of centers serving elementary 
school students reported less difficulty getting information about student needs and the school’s 
curriculum than did directors of centers serving older groups. Difficulties getting this 
information were greater among centers serving middle school students than they were for 
centers serving other age groups. 



Exhibit 4-9 

Percentage of Centers That Reported Challenges in Implementing High-Quality 
Programming, by Type of Challenge and Center Type 
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Source: National Survey, Item 31. 
n = 499 
*p < 0.05 

Exhibit reads: Twenty percent of school-based centers cited lack of information about students’ academic needs 
as a challenge to implementing high-quality programming. 
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Adult Support: Fostering Positive Relationships Between Adults and Students 

One of the most consistent findings about after-school programs is that youth engagement in 
activities depends on positive relationships between adult staff and participants (Birmingham et 
ah, 2005; Carruthers and Busser, 2000; Grossman et ah, 2002; Huang et ah, 2000; Intercultural 
Center for Research in Education & National Institute on Out-of-School Time, 2005; Pierce et al. 
1999). The after-school programs that researchers identified as more successful in offering 
quality programming hire staff who are able to create warm and friendly relationships with 
youth, and provide a challenging and motivating intellectual environment for them (Grossman et 
ah, 2002). Positive staff relationships with students are key to retaining students in a program, 
especially among older youth (Roffman, Papagano and Hirsch, 2001), and one research study has 
linked such positive relationships to positive behavioral adjustment in school (Pierce et al., 

1999). See Appendix F for the observation form that describes the dimensions of adult support. 

In the large majority of observed activities in reading and mathematics (n = 59), staff used 
positive behavior management strategies and showed warm affect toward students, but 
infrequently sought to determine students’ concerns and ideas. In 88 percent of reading activities 
and 84 percent of mathematics activities, staff were observed using positive behavior 
management strategies, such as reinforcement of good behavior (see Exhibit 4-10). However, 
instructors encouraged students to share their ideas in only 30 percent of reading activities and 
21 percent of mathematics activities. 



Exhibit 4-10 

Percentage of Observations in Reading and Math Activities That Demonstrated Student- 
Staff Relationship-Buiiding Efforts, by Relationship-Building Effort and Type of Activity 
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Source: Site visit observations, 
n = 40 for reading; n = 19 for mathematics 

Exhibit reads: In 88 percent of observed reading activities, staff used positive behavior management techniques. 



4. Organizational Supports for Instructional Quality 



52 






Coordination with Supplemental Educational Services 

To strengthen their links with schools, leverage their resources and enhance their services, 
some 21st Century Community Learning Centers coordinated with or became SES providers. 

Fifteen percent of centers were authorized to provide supplemental educational 
services. Each center providing SES did so for an average of 38 students in the core subjects of 
reading and mathematics while, according to the EPICS database, delivering services for a 
median of 100 students across all subjects and activities, including reading and mathematics. Just 
9 percent of all centers reported coordinating with one or two providers, and only 5 percent of all 
centers reported that their coordination activities focused on aligning schedules with the 
providers, while 4 percent indicated they coordinated their academic support activities with the 
providers. 

The vast majority of instructional practices reported for centers providing SES and for 
centers not providing SES were similar. However, SES-providing centers responded differently 
regarding two practices in mathematics and one in tutoring than centers that were not providers. 
The SES providers were less likely than non-SES providers to have students practice basic 
mathematics operations in all or most activities (46 percent versus 68 percent). Eurthermore, 
SES-providing centers were less likely to have students solve mathematical equations in some 
activities than non-SES providers (59 percent versus 85 percent). However, SES providers were 
more likely to offer 2-4 hours of tutoring weekly than non-SES providers (56 percent versus 36 
percent). 

Summary 

Instructional supports for program quality were more widely available in school-based 
centers than in nonschool-based centers. School-based centers paid staff who held teaching 
certificates more than did nonschool-based programs. About one-third of the staff had joined the 
center within the past year. Eow pay and the inability to offer staff full-time positions were 
greater challenges to retention in nonschool-based centers than in school-based centers. With 
respect to coordinating their activities with the regular school day, nonschool-based centers faced 
greater barriers to obtaining information about student needs than did school-based centers. This 
is not surprising, given the beneficial location that school-based centers possess. 

Professional development was not emphasized in the centers. Although more than half of all 
centers reported that they offered paid professional development opportunities to their staff, site 
visit data suggest that those offerings were limited in time and scope. Eour of 12 case study 
centers provided just 1 to 2 days of training or orientation to staff members, and the training 
focused primarily on logistics and program implementation. Content-focused professional 
development is important because it has been linked, in correlational studies, to changes in 
teacher knowledge and practice (Caret et ah, 2001). 
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It was significant that half the centers reported using some form of student- or school-level 
data to improve program practice, given increasing attention to the use of data to monitor 
progress and enhance the quality of before and after-school programs (Yohalem, Wilson- 
Ahlstrom and Yu, 2005). The logic behind these efforts is straightforward: With greater 
accountability for results, educators pay more attention to those results and to their significance 
for ongoing efforts to improve practice. Research in this area, however, is sparse and has not 
been linked to outcomes. 

In nearly all observed activities in reading and mathematics (n = 59), staff used positive 
behavior management strategies and showed warm affect toward students. Such high-quality 
relationships are important because they are linked to greater student engagement with after- 
school programming activities and positive youth outcomes, including improved behavior in 
school (Pierce et ah, 1999). 
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5. Conclusions and Implications for Future Research 

and Evaluation 

Nature and Quality of Reading and Mathematics Activities 

Considering that the previous national evaluation noted the centers’ limited focus on 
academics (Dynarski et ah, 2003; Dynarski et ah, 2004), the fact that nearly all the centers 
funded at the time of this study reported providing activities in reading and mathematics may be 
regarded as a significant shift in approach. The center directors’ responses indicated that reading 
activities were aligned with scientifically based practices: Instruction focused on a broad range 
of skills and employed several strategies for teaching vocabulary, fluency and comprehension. 
Additionally, many centers reported that they asked students to talk or write about something 
they had read. 

Survey findings and observations indicated that mathematics activities may be more limited 
than reading activities. Mathematics instruction focused primarily on two content strands and 
engaged students in tasks with low levels of cognitive complexity. Observation data, while not 
nationally representative, suggest that a more limited range of strategies was used in mathematics 
instruction. Furthermore, in the mathematics activities observed on site visits, students were less 
likely to be on task than were students in reading activities. 

Attendance and Student Participation 

With respect to regular attendance, centers in a nonrepresentative sample reported that 48 
percent of students in centers serving elementary school students attended 60 days or more in the 
last year, or roughly 2 days per week. In this study, 36 percent of middle school students 
attended 60 days or more in the last year. Twenty-eight percent of middle school students in this 
study attended 30-60 days and 36 percent attended fewer than 30 days. In the earlier evaluation, 
20 percent of middle school students attended 5 1 days or more. Attendance was higher than 
reported in the earlier study for school-based centers but not for nonschool-based centers. These 
data also replicated a general trend found in the other studies: Elementary students attend after- 
school activities for more days in the school year than do middle and high school students. These 
data are of limited utility, however, compared with other studies that measured attendance 
directly and distinguished among attendance in different types of programs (summer, before and 
after school). 

Organizational Supports for Instructional Quality 

Two ways that centers’ organizational practices strongly supported instructional quality were 
their use of data and positive staff-student relationships. About half of all centers reported 
making use of data for a variety of purposes, including program evaluation and ongoing review 
of programming activities. Site visitors also observed that staff relationships with students were 
characterized by high levels of warmth and positive behavior management strategies. 

At the same time, professional development was an area that needed greater attention. 
Although more than half of all centers reported offering paid professional development 
opportunities to their staff, site visit data suggested that, in some centers, those opportunities 
were limited in time and scope. 




Centers faced three additional challenges to offering high-quality reading and mathematics 
activities: (a) low pay and lack of benefits were barriers to retaining high-quality staff; (b) 
centers did not always have the information they needed about students’ needs and (c) attendance 
policies did not necessarily result in better attendance rates. 

Key Differences Between Different Types of Centers 

There were few reported differences between school-based and nonschool-based centers’ 
math and reading instruction, except that school-based centers were generally more likely to 
report emphasizing higher-order skills. School-based centers were more likely than nonschool- 
based centers to report that their instructors asked students to make predictions about something 
they were reading, and answer questions, orally or in writing, on something they had read. At the 
same time, nonschool-based centers had students read teacher- selected books more often than 
school-based centers did. In mathematics as well, school-based centers were more likely than 
nonschool-based centers to report emphasizing advanced topics. School-based centers were also 
significantly more likely than nonschool-based centers to ask students to make charts, graphs and 
tables in all or most activities. 

One of the key differences between school-based and nonschool-based centers was the level 
of instructional support for program quality. School-based centers paid staff with teaching 
certificates at higher rates than nonschool-based programs did. For example, school-based 
centers reported that low pay and lack of benefits were less important in retaining staff than they 
were for nonschool-based centers.. With respect to coordinating their activities with the regular 
school day, nonschool-based centers faced greater barriers to obtaining information about student 
needs and performance than did school-based centers. These barriers present formidable 
challenges to nonschool-based centers in partnering with the school-day instructional staff to 
deliver high-quality programming to address individual student needs. 

Significant differences existed in attendance policies and patterns of attendance based on the 
grade level of students served. In centers serving the elementary grades, students were more 
likely to attend regularly; these centers were also more likely to have policies requiring students 
to attend the program for a minimum number of days to remain enrolled. In contrast, centers 
serving high school students were more likely to have a high proportion of students who attended 
fewer than 30 days in 2005-06 than centers not serving that age group. At the same time, centers 
that served middle and high school grades and that had attendance policies reported higher levels 
of attendance than did centers serving these grades that allowed students to participate on a drop- 
in basis. 

Recommendations for Future Research and Evaluation 

Recommended areas of future study include analyzing the role of instructors’ 
qualifications, the quality of reading programming and students’ and families’ needs for after- 
school programs. Finer-grained measures of instructor qualifications may prove more effective in 
predicting differences in instructional quality than the broader measure of teacher certification or 
specialist status. Measures of instructors’ relative confidence in teaching reading and 
mathematics and their knowledge of mathematics teaching may explain these differences (Hill, 
Rowan and Ball, 2005). The current study did not directly measure instructors’ knowledge or 
skills, but future studies could examine those aspects. The content of the professional 
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development provided for center staff could also be considered as part of instructors’ 
qualifications. Future work examining the quality of reading programming could focus on how 
best to capture information on reading when respondents are after-school program administrators 
(rather than school-day administrators) who may be less familiar with specific instructional 
strategies used in academic programming. Future studies concerned with student and family 
demand for after-school programs could collect information on program enrollment capacity and 
waiting lists, as well as reasons why parents or students chose to enroll in after-school programs. 
Effective strategies that centers implement to increase participation in their programs could also 
be analyzed in this regard. 

Last, because all students who take part in 21st Century Community Learning Centers 
programs also participate in learning experiences at home and at school, and some also 
participate in other after-school programs, longitudinal studies that follow participants over an 
extended period could provide a better understanding of how these learning experiences may 
complement one another and enhance academic achievement. Emerging frameworks (e.g., 
Barron, 2006) within the learning sciences that highlight the need to understand how activities, 
resources and interactions outside of school relate to learning within schools could guide such a 
study. Studying how learning experiences in one setting lead students to pursue learning in 
another setting could be the focus of a specific longitudinal study. Connections might be found at 
21st Century Community Learning Centers programs with a strong focus on developing interest, 
knowledge and skill in a particular area and strong connections to the school day and other 
service providers. A 3- to 5-year study of students who participate in such programs could yield 
important new insights into how 21st Century Community Learning Centers programs can 
enhance outcomes for students. 
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Appendix A 

Descriptions of Academic Instruction for 
Centers Serving All Students 




This appendix contains descriptions of academic instruction for all centers, with separate 
descriptions for centers serving middle and high school students. The organization of the 
discussion parallels Chapter 2, which presents data for centers serving elementary school 
students only. Information presented on school-based versus nonschool-based is for all centers, 
including centers that serve elementary, middle or high school students or students at multiple 
levels. Data presented on centers serving middle school students cover any center that included 
middle school students, and data presented on centers serving high school students cover any 
center that served high school students. 




Exposure to Academics After School 

Across all centers, students spent similar amounts of time per week in reading and 
mathematics activities, but they spent more time working on homework and in arts or 
recreation activities. Overall, almost a quarter of centers reported that students participated in 
reading activities and 19 percent of centers conducted mathematics activities for more than 4 
hours per week (see Exhibit A-1). In contrast, higher percentages of centers indicated that 
students participated in both homework (35 percent) and arts or recreation activities (35 percent) 
for more than 4 hours. Because centers were open slightly less than 16 hours per week, on 
average, student participation of 4 hours per week in a particular activity amounted to about 25 
percent of the available time. 



Exhibit A-1 

Percentage of Centers That Reported Participation in Activities by a Typical Student, by 
Type of Activity and Hours of Engagement per Week 




Note; Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 5. 
n = 504 

Exhibit reads: Thirty-five percent of all centers reported that a typical student participated in homework 
activities for more than 4 hours per week. 
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Reading Activities 

Reading Skills Emphasized 

More than half of centers reported that most or all reading activities'^ included the 
skills emphasized by the National Reading Panel (2000). Although phonemic awareness and 
phonics instruction were emphasized in secondary reading activities, centers reported very few 
differences in the skills that were emphasized in different grade levels. The largest percentage of 
centers serving middle schools reported emphasizing comprehension and vocabulary in all or 
most activities (65 percent and 62 percent, respectively). In particular, there was a greater-than- 
expected emphasis on phonics and phonemic awareness at the secondary level. Somewhat 
surprisingly, 39 percent reported focusing on phonics skills, and 37 percent reported focusing on 
phonemic awareness in all or most activities (see Exhibit A-2). 



Exhibit A-2 

Percentage of Centers Serving Middle School Students That Reported Reading Activities 
Emphases, by Reading Skill Emphasized and Inclusion Level 
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Source; National Survey, Item 22. 
n = 315 

Exhibit reads; Sixty-five percent of centers serving middle school students reported emphasizing comprehension 
in most or all enrichment activities in reading. 



Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. 
Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment 
activities, but not homework assistance. 
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Centers serving students in high schools revealed few differences from centers serving 
younger students in the reading skills being emphasized. The largest percentage reported 
stressing comprehension and vocabulary in all or most activities (74 percent and 71 percent, 
respectively) (see Exhibit A-3). Although teaching phonics and phonemic awareness are 
typically not part of local and state standards at the secondary level, a large percentage of centers 
reported emphasizing those skills in their academic activities. Forty-five percent of centers 
serving high school students reported focusing on phonics in all or most reading activities, and 
39 percent of centers serving this age group stressed phonemic awareness. The study team did 
not collect site visit data in high school-serving centers, so the reported data cannot be confirmed 
or challenged by observations. Nonetheless, the study team finds it likely that high school 
directors overreported the frequency with which academic activities focused on these skills. 
Another possible explanation for the pattern is that students in these high schools struggled with 
the most basic of reading skills; enrichment targeted to them reflected their needs. 



Exhibit A-3 

Percentage of Centers Serving High School Students That Reported Reading Activities 
Emphases, by Reading Skill Emphasized and Inclusion Level 
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Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 22. 
n= 107 

Exhibit reads: Seventy-four percent of centers serving high school students reported emphasizing comprehension 
in most or all enrichment activities in reading. 



Use of Instructional Strategies for Teaching Vocabulary, Fluency and 
Comprehension 

Although 99 percent of all centers reported using at least one strategy in reading, the 
most commonly reported reading activities were designed to foster more independent 
reading and greater comprehension. About half of centers (46 percent) reported that they 
asked students to explain or support statements about something they read in all or most reading 
activities. About a third of centers had students talk about or write answers to questions on 
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something they had read (39 percent), asked students to help other students understand new 
words (35 percent) and asked students to make predictions about a text they are reading (35 
percent) (see Exhibit A-4). 



Exhibit A-4 

Percentage of Centers That Reported Student Participation in Instructional Strategies in 
Reading Activities, by Strategy and Inclusion Level 




Note; Percentages may not equal 100 percent due to rounding. 

Source; National Survey, Item 21. 
n = 501 

Exhibit reads; Forty-six percent of centers reported having students explain or support their understanding of what 
they had read in most or all reading enrichment activities. 

Reports of instructional strategies differed by grade level and center type. Centers serving 
elementary students were less likely than centers serving other grade levels to have students talk 
with peers about what they read and more likely to have students explain or support something 
they read. These centers were also more likely to have students make prereading predictions. 
Centers serving middle school grades were less likely than those serving other grades to have 
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students explain or support something they read. All these differences are consistent with 
differences in reading skill and are most likely associated with student grade level. School-based 
centers were more likely than nonschool-based centers to report that their instructors asked 
students to make predictions about something they were reading. At the same time, nonschool- 
based centers had students read teacher- selected books more often than did school-based centers. 

Mathematics Activities 

Mathematics Skills Emphasized 

Among centers serving middle school students, 64 percent reported that they emphasized 
operations with whole numbers (such as addition, subtraction, multiplication and division) in all 
or most mathematics activities (see Exhibit A-5). About half targeted number sense in all or most 
mathematics activities (53 percent). Centers serving middle school students gave less attention to 
advanced content than to operations with numbers and other content: 18 percent emphasized 
algebraic concepts, 14 percent focused on geometry and 1 1 percent focused on data analysis in 
all or most activities. 



Exhibit A-5 

Percentage of Centers Serving Middle School Students That Reported Emphases in Math 
Activities, by Type of Content Emphasized and Inclusion Level 




Note: Percentages may not equal 100 percent due to rounding. 

Source: National Survey, Item 24. 
n = 314 

Exhibit reads: Sixty-four percent of centers serving middle school students emphasized operations with whole 
numbers in most or all mathematics enrichment activities. 
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Among centers serving high school students, 69 percent reported that they emphasized 
operations with whole numbers in all or most mathematics activities (see Exhibit A-6). Fifty- 
eight percent stressed number sense in all or most activities. Geometry and data analysis received 
the least amount of attention among centers serving high school students: 20 percent focused on 
geometric concepts and 16 percent emphasized data analysis in all or most activities. 



Exhibit A-6 

Percentage of Centers Serving High School Students That Reported Emphases in Math 
Activities, by Type of Content Emphasized and Inclusion Level 
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Source: National Survey, Item 24. 
n= 117 

Exhibit reads: Sixty-nine percent of centers serving high school students emphasized operations with whole 
numbers in most or all mathematics enrichment activities. 



Across all grade levels, school-based centers were more likely than nonschool-based centers 
to emphasize advanced topics in mathematics. In particular, school-based centers were 
significantly more likely than nonschool-based centers to give at least some emphasis to 
geometry (86 percent versus 72 percent), algebra (79 percent versus 65 percent) and data 
analysis (74 percent versus 62 percent). For foundational topics such as number sense and 
operations, however, both types of centers had similar emphases. 
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Use of Instructional Strategies for Teaching Mathematics 

The observed mathematics activities in the learning centers corresponded to instructional 
strategies. In particular, the incorporation of low-complexity cognitive tasks such as practicing 
basic mathematics facts as well as high-complexity tasks such as multistep problem solving in 
the learning centers was aligned with research identifying these methods as effective 
instructional strategies (Porter, 2002). Similarly, providing students the opportunity to explain 
the reasoning behind a problem solution helps to build proficiency in computation and 
translation of word problems. It also helps teachers to ensure that students have the conceptual 
framework necessary for understanding the grade-level mathematics (National Mathematics 
Advisory Panel, 2008). Experimental research studies have shown that computers and graphing 
calculators have a positive effect on mathematics achievement (Dixon et ah, 1998). 

The National Mathematics Advisory Panel (2008), which was established to promote 
American students’ knowledge of mathematics and improve their performance in the subject, 
concluded that cooperative learning has significant effects on student outcomes, particularly in 
teaching young children mathematical operations. Strategies include team-assisted 
individualization and peer-assisted learning strategies in which heterogeneous students are 
grouped so that those who are weak in a specific skill can work with others who are strong in 
that area. Additional evidence suggests that teaching mathematics from a real-world perspective 
positively affects student academic achievement. 

Although 98 percent of all centers reported using at least one strategy, the most 
common strategies used in all or most mathematics activities were practicing mathematical 
facts (63 percent) and asking students to explain their reasoning in solving a problem (47 
percent). Twenty-three percent of all centers reported having students use computers to support 
mathematics activities or to solve open-ended problems in mathematics (see Exhibit A-7). 
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Exhibit A-7 

Percentage of Centers That Reported Student Participation in Instructional Strategies in 
Math Activities, by Strategy and Inclusion Level 




Note; Percentages may not equal 100 percent due to rounding. 

Source; National Survey, Item 23. 
n = 501 

Exhibit reads; Sixty-three percent of all centers reported asking students to practice basic mathematics facts in most 
or all instructional activities in mathematics. 

Mathematics activities differed in certain respects among center types and the grade levels 
served. For example, school-based centers were significantly more likely than nonschool-based 
centers to ask students to make charts, graphs and tables in all or most activities (20 percent 
versus 10 percent). Centers serving elementary school students were less likely than centers 
serving older groups to report using graphing calculators in all or most activities. 

Instructor Education and Experience 

Centers reported that more than half of staff providing instruction in reading and 
mathematics had prior experience as certified classroom teachers. Fifty-five percent of the 
staff who provided instruction in reading or mathematics had been regular classroom teachers, 
and 23 percent had been instructional specialists in reading or mathematics. Twenty-three 
percent of staff providing instruction in reading or mathematics had served as classroom aides. 
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Centers reported that nearly two-thirds of reading and mathematics instructors had a 
bachelor’s degree or higher (64 percent for reading and 64 percent for mathematics). A 

third of instructors reported having a bachelor’s degree or a bachelor’s degree plus some 
graduate training (36 percent for reading and 37 percent for mathematics). 

Feedback to Students and Parents on Student Progress 

The majority of centers providing academic instruction reported providing informal 
feedback to students; written or formal feedback was less common. For instance, across all 
grade levels, 86 percent of school-based centers reported providing verbal feedback to students 
on work assigned by the school and 85 percent provided feedback on their behavior. In contrast, 
about a quarter of centers provided students written feedback on work assigned by the school (23 
percent) and on work assigned in the center (25 percent). Nonschool-based centers were 
significantly more likely to provide written feedback on work assigned in the center than were 
school-based centers (35 percent versus 25 percent) (see Exhibit A-8). 



Exhibit A-8 

Percentage of Centers That Provided Feedback About Their Progress, by Type of 

Feedback and Center Type 



Feedback 


Center Type 


Nonschool- 

based 


School- 

based 


Verbal feedback from center staff: 


On work assigned by the school 


81% 


86% 


On work assigned at the center 


76% 


67% 


On student behavior 


86% 


85% 


Written feedback from center staff: 


On work assigned by the school 


25% 


23% 


On work assigned at the center 


35%* 


25%* 


On student behavior 


44% 


44% 


“Points” or rewards for program accomplishments 


66%* 


48%* 


Certificates or awards for accomplishments 


74%* 


62%* 



Source: National Survey, Item 18. 



n = 500 
*p < 0.05 

Exhibit reads: Eighty-six percent of all school-based centers provided verbal feedback to students on work assigned 
by the student’s school. 

Centers serving elementary school students were significantly more likely than centers 
serving older students to report providing verbal feedback to students on work assigned at the 
center. Seventy-two percent of centers serving elementary school students reported providing 
this type of feedback, compared with 66 percent of middle school centers and 67 percent of high 
school centers. Significantly more centers serving elementary school students than centers 
serving other age groups also reported providing written feedback on student behavior. 

A majority of centers reported providing frequent, informal feedback about student 
progress to parents. More than half (55 percent) of centers reported talking with parents nearly 
every week when the parents picked up or dropped off their child. Five percent sent home 
written reports of students’ progress in the after-school program nearly every week, but 54 
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percent never did so or did so rarely. Similarly, 2 percent of centers held parent-center staff 
conferences nearly every week, and 56 percent never or rarely held conferences. There were no 
substantive differences by grade level served or type of center. Elementary-serving centers were 
more likely than centers not serving this age group to provide feedback to parents about student 
behavior. Nonschool-based centers were more likely than school-based centers to provide 
feedback on work assigned and to use points and certificates for accomplishments. 
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Appendix B 

Case Study Observations 




This appendix presents analyses conducted by coders working independently to characterize 
organizational goals and practices of 21st Century Community Learning Centers programs 
visited in fall 2006 and spring 2007. The focus of the analysis is identifying and describing 
practices in after-school programming, and identifying the factors that influence those practices. 
The observation instrument used to collect this data is available in Appendix F. 

Case Study Observations: Identifying Practices 

Alignment of Observed Enrichment Activities With Practices 

Observed enrichment activities were partially aligned with practices. Analysts rated 
enrichment activities observed during the site visits for their alignment with practices in four 
broad domains: content, instruction, engagement and relationships. For each of these domains, 
the observed activities were rated on a four-point scale. Raters assigned one point if the observed 
activity did not occur, two points if the activity focused on the feature but there was no 
alignment, 3 points if the activity focused on the feature and there was partial alignment and four 
points if the feature was a focus and there was good alignment with the definition of practice. 

The average ratings of quality were highest in engagement and relationships (see Exhibit 
B-1). The typical activity was partially aligned in terms of engagement (average rating = 3.6) and 
relationships (average rating = 3.3). There was more limited alignment in content (average rating 
= 2.9) and instruction (average rating = 2.7). 

Although different from classroom instruction, enrichment programming in the after-school 
setting maintains connections across learning environments, integrates learning goals and 
deepens exploration and skill acquisition for children (Noam, 2003). In addition to focusing on 
academics, the observed enrichment activities actively engaged learners in constructing 
knowledge and building skills, and targeted particular social or personal skills. Although it is not 
possible to generalize the case study findings to the entire population of 21st Century 
Community Learning Centers, these findings suggest that centers are employing practices and 
approaches that are aligned with research on program quality. 
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Exhibit B-1 

Ratings of Observations in Enrichment Activities 




12 3 4 

Average Rating 



Source: Observation Activity Rating Rubric. 
n = 51 

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 
Exhibit reads: The level of student engagement for enrichment activities was rated 3.6 by observers in terms 
of alignment with practice. 

Reading and Mathematics Activities 

Alignment of Observed Reading Activities With Practices 

Analysts rated the reading activities observed during the site visits for their alignment with 
practices in four broad domains: content, instruction, engagement and relationships. For each of 
these domains, the observed activities were rated on a four-point scale as described earlier. The 
“content” and “instruction” categories are quality ratings made by observers and not simply 
occurrences of content. 

Observed reading activities rated high in terms of student engagement (see Exhibit B-2). 
The typical activity had good alignment in engagement (average rating = 3.8) and partial 
alignment in relationships (average rating = 3.3) and content (average rating = 3.3). There was 
more limited alignment with instructional practices (average rating = 2.9). 
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Exhibit B-2 

Ratings of Observations in Reading Activities 



Engagement 
^ Relationships 

O 

'•I— > 

o 

ro 

dI Content 

Instruction 



12 3 4 

Average Rating 




Source: Observation Activity Rating Rubric. 
n = 38 

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 
Exhibit reads: The level of student engagement for reading enrichment activities was rated 3.8 by 
observers in terms of alignment with practice. 



Mathematics Activities 

Alignment of Observed Mathematics Activities With Practices 

Analysts rated the mathematics activities observed during the site visits for their alignment 
with practices in four broad domains: content, instruction, engagement and relationships, as 
described earlier. The “content” and “instruction” categories are quality ratings made by 
observers and not simply occurrences of content. 

The content of observed mathematics activities aligned with practice (see Exhibit B-3). 
The typical activity had good alignment in content (average rating = 3.7) and partial alignment in 
engagement (average rating = 3.3) and relationships (average rating = 3.1). Alignment was more 
limited in instruction (average rating = 2.9). 
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Exhibit B-3 

Ratings of Observations in Mathematics Activities 




Source: Observation Activity Rating Rubric. 
n= 19 

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 
Exhibit reads: The level of student engagement for mathematics enrichment activities was rated 3.3 by 
observers in terms of alignment with practice. 
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Example of a Mathematics Activity 

In a mathematics activity that scored a 4 on the content and engagement observation rating scaies, 
indicating good aiignment with practices, a center used computerized instruction aiigned with the 
state’s content standards. The program, which was aiso used during the schooi day, inciuded a 
continuing assessment for students so that they were directed to new skiiis as they successfuiiy 
mastered concepts, it aiso prompted students to caii on a teacher when they repeated errors and 
couid benefit from direct instruction. 

in the observed activity, after-schooi students worked at the computer under the supervision of their 
reguiar ciassroom teacher. Students worked at different skiii ieveis on different topics, inciuding 
number sense, expanded notation, measurement and piace vaiue. As they worked independentiy, the 
teacher circuiated, expiaining math concepts and procedures to individuai students as necessary. 
When students mastered objectives, the program presented new materiai to them, if they stumbied, 
the computer aierted the teacher’s diagnostic screen, and the teacher moved to the struggiing 
students with direct assistance. When students faitered, a pop-up “Teacher Time” window appeared, 
and the student summoned the teacher to diagnose the error. The teacher, together with the 
computerized program, kept students on task and moving swiftiy through the instruction. When 
students made mistakes, the teacher assured them that “making mistakes is how we iearn the most.” 



In summary, these findings suggest that reading activities address a broader range of 
processes than mathematics activities. Although reading instruction across center types focused 
on each of the reading processes emphasized by the National Reading Panel, the content strands 
of mathematics were not well represented in center activities. From the observation data, 
instructional strategies of mathematics activities appeared to be limited, suggesting a narrow 
range of instruction. Furthermore, the cognitive demand of the reading activities observed — both 
as part of formal instruction and in enrichment activities — was generally higher than the demand 
in activities where mathematics was explicitly taught or embedded. This trend was especially 
evident among nonschool sites, which also had fewer certified teachers offering instruction. 
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Appendix C 

Sampling Approach and Weighting of Surveys 




Center Director National Survey 

Intended Survey Respondents 

The national survey was administered to the center director in charge of after-school services 
at sampled sites. A “site” was generally defined as a location where 21st Century Community 
Learning Centers services were offered. 

Sampling Approach 

The evaluation team used the Profile and Performance Information Collection System 
(PPICS) database, an online data collection and reporting system that collects grantee- and 
center-level data, to design the sample. To ensure adequate representation across characteristics 
of interest, the sample was stratified by the school levels the center served (elementary, 
secondary or both), the center location (school-based or nonschool-based) and the percentage of 
students eligible for free or reduced-priced lunch. The team identified 5,122 centers that met the 
following survey population-inclusion criteria: 

1 . At least one center was expected to be funded for the duration of the study. 

2. The center offered academic enrichment services in mathematics, reading and 
technology. 

3. The center had been in operation for at least 1 full school year by the time of the survey. 

A total of 4,347 centers did not meet the survey population-inclusion criteria. Although nearly all 
sites offered reading and mathematics instruction, 19 percent of centers did not offer technology 
services and thus were excluded from the sample. 

Degree of Precision 

A simple random sample of 600 centers with 516 respondents allowed 95 percent confidence 
intervals to be developed for proportions (e.g., the proportion of centers that had organized 
homework activities) with a half- width of no more than 4.3 percentage points. However, 
because of the need to weight the sample to account for nonresponse, the target “effective” 
sample size was approximately 407, or about 79 percent of the number of respondents. That is, 
the 516 respondents for this study provided the same precision of estimation as 407 respondents 
who were selected via a simple random sampling method and were equally weighted. Thus, the 
95 percent confidence intervals were designed to have a half- width of approximately 4.9 
percentage points. If two approximately equally sized groups of centers were compared, 
differences in the means group would have a 95 percent confidence interval no larger than 9.7 
percentage points. That is, descriptive differences between groups of more than 9.7 percentage 
points would be large enough to be practically important and statistically significant. 



If the proportion of the sample with a certain characteristic is greater than or less than 50 percent, the half-width of 
the confidence interval is less than 4.3 percentage points. For example, for a proportion of 10 percent or 90 
percent, the half-width would be 2.6 percentage points. 




The effect size that is detectable with 80 percent power (using a two-sided 5 percent alpha 
level) is a function of the effective sample size for each subgroup. (In general, the study team 
expected a design effect of about 1.2 corresponding to an effective sample size that was about 85 
percent of the actual number of respondents.) 

Selecting the Sample 

From the initial pool of 5,122 centers, the study team selected a random sample of 600 
centers stratified on school-based or nonschool-based centers (with oversampling of the 
community-based centers to derive more precise estimates about the practices of those centers). 

Response Rates 

During follow-up communication with survey nonrespondents, the project directors 
determined that 43 centers should be eliminated from the sample: 1 1 centers were eliminated 
because they had been defunded; 21 because they had not received program funds, did not serve 
the K-12 population or had no center at the school address supplied; 6 because they were 
duplicates of other centers and 4 because all efforts to obtain correct contact information failed. 
Thus, the size of the sampling frame after eliminating ineligible centers was 558 for the national 
survey. For the national survey, 516 surveys were completed, resulting in a response rate of 92.5 
percent. 

Weighting 

The national survey sample was weighted to reflect the stratified design. However, to be able 
to make statements about the national population of centers, the data were poststratified to reflect 
the full population of eligible centers. The study team also used Deming’s procedure to weight 
the respondents so that the weighted marginal counts were the same as in the universe of sites 
that were funded at the time of the study and that had been in operation at least a year (e.g., the 
weighted number of responding centers serving elementary school students was the same as the 
number of centers serving elementary school students in the universe). The specific 
characteristics used to poststratify the respondents came from the PPICS database and from U.S. 
Census data ZIP code files. The specific variables used from the PPICS database were the 
numbers of students served, program longevity, prior history with offering academic services, 
the primary grade levels served and the number of weeks per year that centers were open. Data 
from the U.S. census ZIP code files used to poststratify respondents were the percentage of a ZIP 
code’s population living in an urbanized area and the percentage of minority population in that 
area. 

In addition, to correct for biases introduced by differences among some subpopulations (e.g., 
elementary and high schools may have different response rates), the evaluation team 
poststratified the respondents on the basis of characteristics available in the sampling frame and 
used Deming’s procedure to weight the respondents so that the weighted marginal counts were 
the same as in the universe (e.g., the weighted number of responding centers serving elementary 
school students was the same as the number of centers serving elementary school students in the 
universe). 
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Center Director Focused Telephone Survey 

Intended Survey Respondents 

The focused telephone survey targeted the center director (the same respondent to whom the 
national survey was sent) as the respondent. In addition to asking for information from the center 
director, the telephone survey asked a series of questions designed to elicit information about 
three staff members. The research team supplied the center director with an algorithm for 
randomly selecting appropriate staff members as targets for the questions in the survey. 

Sampling Approach 

Determining the Sample Size 

To obtain a response rate of 85 percent, 140 centers were selected to receive a focused 
telephone survey along with the national mail survey. Of these 140 centers, 116 completed both 
surveys. 

Degree of Precision 

A simple random sample of 140 centers with 122 respondents would allow 95 percent 
confidence intervals to be developed for proportions (e.g., the proportion of centers that had 
organized homework activities) with a half- width of no more than 9.0 percentage points. 
However, because of the need to weight the sample to account for stratification (into school- 
based and nonschool-based strata) and differential nonresponse between the two types of centers, 
the target “effective” sample size was approximately 84 or about 69 percent of the number of 
respondents. That is, the 122 respondents for this survey would provide the same precision of 
estimation as 69 respondents who were selected via a simple random sample and were equally 
weighted. Thus, the 95 percent confidence intervals would have a half- width of approximately 
12.0 percentage points. If two approximately equally sized groups of centers were compared, 
differences in the proportion of centers in each group with a certain characteristic would have a 
95 percent confidence interval no larger than 24.3 percentage points. That is, differences of 
more than 24.3 percentage points would be considered statistically significant. Thus, differences 
that were large enough to be practically important would also be statistically significant. 

Selecting the Sample 

The evaluation team used a sequential sampling approach (similar to that described for 
selecting the national mail survey sample) to select the sample of centers that were surveyed 
using the focused telephone questionnaire. 



These results are the consequence of general statistical principles. They would apply to any survey with 122 
respondents and a sampling efficiency of 69 percent. Even if the sampling efficiency were 100 percent, the half- 
width would be (at most) 17.4 percentage points. Sampling efficiency could be as high as 100 percent, but would 
not be known until the responses were obtained (allowing us to ascertain the characteristics of respondents versus 
nonrespondents relative to information known about the universe). 
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The focused telephone survey collected data on staff qualifications for up to three staff 
members for each center. Statisticians used all the responses about staff in calculating staff 
characteristics. Because each center nominated three staff on average, no center effect had to be 
accounted for in the data. 

To select the target staff members, the evaluation team asked the center director to provide a 
list of staff whose primary responsibilities were planning and delivering academically oriented 
content to students. From that list, the evaluation team randomly selected three individuals and 
asked the center director to provide background information about their roles, credentials and 
qualifications. 

Response Rates 

During follow-up procedures with nonrespondents, the project directors determined that 8 of 
the 140 centers were ineligible for inclusion. Thus, the size of the sampling frame after 
eliminating ineligible centers was 132 for the focused survey. For the focused survey, 122 
surveys were completed, resulting in a response rate of 92.4 percent. A total of 1 16 centers 
completed both a national and a focused survey, for a response rate of 87.8 percent. 

Weighting 

The telephone survey sample was weighted to reflect the stratified design. In addition, the 
response rate among some subpopulations differed (e.g., elementary and high school centers had 
different response rates). To correct for biases that such differences could introduce, the 
statisticians poststratified the respondents on the basis of characteristics available in the sampling 
frame and used Deming’s procedure to weight the respondents so that the weighted marginal 
counts were the same as in the universe (e.g., the weighted number of responding centers serving 
elementary school students would be the same as the number of centers serving elementary 
school students in the universe). 
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Case Study Site Visits 

Purpose of the Site Visits 

The purpose of the case study data collection and analysis in fall 2006 and spring 2007 was 
to find after-school centers whose students’ academic outcomes had improved — “promising 
sites” — compare them with more typical sites, examine the activities and practices of the 
promising centers and make their successful strategies accessible to others. We visited 12 sites at 
elementary and secondary schools. Of particular interest to the Department of Education for the 
site visits was consideration of the links between the degree of academic focus and academic 
outcomes, alignment of student needs with academic activities and the links between schools and 
after-school centers. Overall, the analysis revealed few differences between promising and 
typical sites across the dimensions examined. The analysis used to distinguish between 
promising and typical sites did not provide sufficient warrant for making judgments about which 
goals, strategies and organizational practices were associated with more promising results. As a 
result, the case studies were analyzed with respect to how each case aligned with policy or 
research on best practices. Cross-case analyses were conducted to illustrate the different ways in 
which sites were aligned or seeking to align with policy and research. 

Intended Respondents 

The site visits included at least three interviews with center staff and school personnel (i.e., 
with the center director, the lead instructor at the after-school center and at least one school staff 
member linked to the center). The site visits also included observations of a minimum of six 
after-school activities at the centers, with primary focus on reading and mathematics activities. 
Finally, as part of the site visit, the researchers collected relevant center artifacts (e.g., schedules, 
student worksheets) and existing student academic achievement data to assess whether test 
scores and academic skills were improving (e.g., data collected by the after-school center, feeder 
schools’ Adequate Yearly Progress status). 

An initial planning conversation with the center director helped to identify the appropriate 
individuals for site visit interviews, and the study team developed interview guides for four types 
of staff associated with each center: 

Center director. The center director was the best-placed individual to have salient 
information about the operations and organization of learning at a 21st Century Community 
Learning Centers program. 

Lead center instructor. With the assistance of the center director or main contact, the site 
visitors selected a key staff member who provided academically oriented programming at the 
center. 

School staff. With the assistance of the center director, the site visitors identified a staff 
member from the school who was familiar with the center’s after-school program and/or who 
provided coordination between the school and the center. 
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Grant director. In most cases, 21st Century Community Learning Centers have more than 
one location where services are offered, and a grant director coordinates the overall efforts of 
those locations. In those cases, the site visitors interviewed the grant director. When the center 
director and grant director were the same person, the site visitors combined the two interview 
protocols to avoid asking duplicate questions. 

Selecting Case Study Sites 

A top priority in this study was to identify and visit centers that had a strong academic focus 
as evidenced by their student outcomes and other components of practice. Because survey data 
were not available to select cases, data reported in the PPICS database provided the best 
evidence about student outcomes for use in selecting promising sites to visit. Initially, the study 
team selected the following three data elements from PPICS as the primary criteria in this 
selection process: 

1. Percentage of students whose reading and/or mathematics assessment scores increased as 
of the 2003-04 reporting period 

2. Number of hours in reading and/or mathematics instruction per week 

3. Student attendance at a center (for at least 30 days in the 2003-04 reporting period) 

Centers with these promising characteristics were selected as a subset of the stratified sample 
of centers to be surveyed. For the entire sample of centers in the PPICS database, the researchers 
first identified, separately for reading and for mathematics, a mean and standard deviation of 
percentage of students whose scores increased. Next, the researchers identified as “promising” 
two samples of centers: one in which the percentage of students was at least 1 standard deviation 
above the mean for reading, and a second in which the percentage of students was at least 1 
standard deviation above the mean for mathematics. For sites in the promising category, the 
evaluation team then eliminated sites that offered fewer than 5 hours of instruction in its category 
(reading or mathematics) and that indicated that fewer than 75 students had attended for at least 
30 days in the previous year. For each promising site, the researchers also identified a paired site 
to serve as a more typical one for purposes of comparison. Typical sites had to have similar 
ethnic diversity among students; be in a locale of similar size and in the same state and be within 
1 standard deviation, plus or minus, of the average reported percentage of students whose test 
scores increased. Thus, each pair had two sites serving relatively similar populations, but with 
different levels of test score improvement. 

Sorting against these three criteria resulted in a subset of possible centers from which sites 
were selected for further screening. From this subset, the evaluation team identified a pool of 40 
sites for further screening: 10 pairs of reading sites and 10 pairs of mathematics sites. 
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Telephone Screening Process 

The next step in the site selection process consisted of conducting telephone screening 
interviews with center directors from the identified centers. In the interviews, trained callers 
collected information about each site’s availability for a site visit and validated the PPICS data 
used to identify the site initially. The callers gathered information about the center’s academic 
goals and activities and about its use of data for assessing program quality and student outcomes. 
They also explored instructional strategies, student participation, links with the school day, 
staffing, student-instructor relationships and staff development. Callers entered the responses 
directly into a database developed for this purpose. Research analysts summarized these data and 
used them to develop a one-page center profile for each of the initial pairs of sites contacted for 
presentation to the Department as candidate sites. 

Joint Review and Final Selection Process 

The research team used the criteria described above to create a subset of centers to visit. 
However, many of the sites proved to be in the same state. To ensure a geographically dispersed 
group of sites, the project directors selected no more than one pair of sites for each subject area 
in a single state. This additional criterion reduced the ability to use the telephone screening data 
to select sites; the researchers then prepared a set of recommended sites on the basis of their 
availability and the initial PPICS screening for the Department to review. Twelve centers were 
selected for site visits. 

Site Notification Process 

The Department contacted all the primary sites by letter and by telephone, inviting their 
participation in the case studies. Within a week of mailing the letters, the study team confirmed 
by telephone whether or not the center was willing to participate in the site visits. If a center 
declined to participate in the case studies, the researchers contacted a corresponding alternate 
center immediately to seek to replace the nonparticipating center. 



18 



Florida has two pairs of sites in the sample because the research team had visited one pair of sites in Florida and 
one in California before modifying the selection criteria. 
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Appendix D 
Survey Instruments 




0MB No. 1875-0239 Approval Expires 10/31/07 



21st Century Community Learning Centers Program Quality Study 



Center Director National Survey 



Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to 
respond to a collection of information unless such collection displays a valid 0MB control number. The valid 0MB 
control number for this information collection is 1875-0239. The time required to complete this information 
collection is estimated to average 1.5 hours per response, including the time to review instructions, search existing 
data resources, gather the data needed, and complete and review the information collection. If you have any 
comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: 
U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns regarding the 
status of your individual submission of this form, write directly to: Margery Yeager, Policy and Program Studies 
Service, U.S. Department of Education, 400 Maryland Ave, S.W., Room 6W213, Washington, D.C. 20202. 






Center Director NATIONAL Survey 
March 2006 



Note to Survey Recipient: 




This survey should he completed primarily hy the director of the 21st Century Community 
Learning Center. Most of the questions can he easily answered hy or are specific to the center 
director. However, we encourage you to confer with appropriate staff members for questions 6-7 
and 21-26 of this survey and with any other appropriate staff members for other items that may 
be difficult for you to answer. Answer all questions in the survey only for the center indicated 
in the cover sheet to the survey. 


Topics addressed in the survey include: 




Overall Description of Program 
Program Goals 
Program Approach 
Center Structure and Policies 
Tracking Outcomes 
Student Academic Support 

Reading Support Activities 
Mathematics Support Activities 
Use of Technology Resources 
Staffing 

Staff Pay and Benefits 
Staff Retention 




Please record the names and titles of all respondents in the space helow. 


Name(s)/Title(s) of the person(s) responding to 
this survey: 


Contact information (telephone or e-mail): 


Primary Respondent 

Name: 


Telephone: 


Title: 


E-mail: 


Respondent for Items 6-7, 21-26 

Name: 


Telephone: 

E-mail: 


Title: 




Other Respondents: 

Name: 

Title: 
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IMPORTANT NOTE: 




Please use a BLACK pen. Blue or red pens and pencil cannot be read by our scanners. 


/7 


When asked to mark boxes, make an "X" through the box. 


{/ 


Sample: Right H Wrong 0 




Use block printing when you complete any text or numeric responses. If you wish to change 




a response, please mark the correct response and CIRCLE it. 



Overall Description of Program 

Program Goals 

1. To what extent is each of the following an objective or goal of the programs at your center? 
(Mark ONE in each row.) 





Not an 
objective 


Minor 

objective 


Major 

objective 


a. Provide a safe environment for students 


□ 


□ 


□ 


b. Help students improve their academic performance 
(e.g., grades, test scores) 


□ 


□ 


□ 


c. Help students develop socially 


□ 


□ 


□ 


d. Provide opportunities for cultural enrichment 


□ 


□ 


□ 


e. Provide recreational activities for students 


□ 


□ 


□ 


f. Provide hands-on activities to supplement academic 
knowledge learning 


□ 


□ 


□ 


g. Help parents and/or other adults with literacy or 
other skills (e.g., parenting) 


□ 


□ 


□ 


h. Help connect students to their community through 
service or other special projects 


□ 


□ 


□ 


i. Identify health or social services that students need 


□ 


□ 


□ 


j. Provide students with positive adult guidance and/or 
mentors 


□ 


□ 


□ 
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2a. Which of the goals or objectives listed in question 1 are your center’s three MOST 
IMPORTANT ones? 

(Select THREE goals or objectives from question 1 and indicate their importance by 
marking the boxes under their corresponding letters in the rows below. Mark ONE in each 
row.) 



abcdef ghi j 



Most important goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Second most important goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Third most important goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



2b. Which of the goals or objectives listed in question 1 do evaluation data show your center is 
making the MOST PROGRESS in achieving? 

(Select THREE goals or objectives from question 1 and indicate your progress toward them 
by marking the boxes under their corresponding letters in the rows below. Mark ONE in 
each row.) 



abcdef ghi j 



Highest level of progress made 
for this goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Second highest level of progress 
made for this goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Third highest level of progress 
made for this goal 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



3. What indicators does your center use to measure your program’s success? (Mark ALL that 
apply.) 

□ a. Staff observations of youth 

□ b. Student questionnaires 

□ c. Parent questionnaires 

□ d. Teacher reports 

□ e. Assessments administered to students as part of the before- or afterschool program 

□ f. Student grades 

□ g. Student achievement test scores 

□ h. Other (specify): 
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Program Approach 

For the following questions, answer for the most recent program year in your program. 

4a. Our program year began: 4b. And it ends or ended: 

□ Summer 2005 □ Spring 2006 

□ Fall 2005 □ Summer 2006 

□ Winter 2005 



5a. On average, for how many hours per week does a typical student participate in activities at 
the center? 

hours 



5b. On average, for how many hours per week does a typical student participate in the following 
activities at your center? (Mark ONE in each row.) 





Not 

offered 


No 
more 
than 2 
hours 


2-4 

hours 


5-6 

hours 


More 
than 6 
hours 


a. Homework help in a group setting 


□ 


□ 


□ 


□ 


□ 


b. Tutoring (one-to-one or peer) 


□ 


□ 


□ 


□ 


□ 


c. An academic activity in which reading 
instruction or practice takes place (not 
tutoring) 


□ 


□ 


□ 


□ 


□ 


d. An academic activity in which 

mathematics instruction or practice takes 
place (not tutoring) 


□ 


□ 


□ 


□ 


□ 


e. An arts or recreation enrichment activity 
(e.g., sports, outdoor games, crafts, 
theater, music) 


□ 


□ 


□ 


□ 


□ 


f. A science activity, project, or science- 
related field trip (e.g., to a local science 
museum) 


□ 


□ 


□ 


□ 


□ 


g. A service or other supplemental activity 
(e.g., counseling, drug and violence 
prevention, mentoring) 


□ 


□ 


□ 


□ 


□ 


h. An activity that combines two or more of 
the types of activities listed above. 


□ 


□ 


□ 


□ 


□ 



(Specify letters of the items: □ □ ) 
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6. What strategies do you use to help meet students’ academic goals? (Mark ALL that apply.) 

The center I direct seeks to help students meet schools’ academic goals by: 

□ a. Adopting the same textbooks and strategies as used during the school day to help 

students with the academic subjects they find difficult 

□ b. Supplementing subject-matter instruction with exploratory activities, field trips, 

and/or applied learning 

□ c. Providing arts, physical fitness, field trips, and active enrichment learning 

opportunities that our feeder schools (i.e., schools students we serve attend) no longer 
offer 

□ d. Providing opportunities for students to participate in subjects not traditionally offered 

in schools 

□ e. Providing leadership and/or community service opportunities for students 

□ f. Helping parents become more involved in their children’s education 

□ g. Providing students with a place where they have freedom to talk about their academic 

difficulties in a safe environment 

□ h. Other 

(specify) : 



7. Which of the above statements BEST describes your approach to helping students meet their 
academic goals? (Mark ONE.) 



a 


b 


c 


d 


e 


f 


g 


h 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



Center Structure and Policies 

8. For how many years have before- or afterschool services been offered at this particular center 
(including before the beginning of the 21st Century Community Learning Centers grant)? 
(Mark ONE.) 

□ a. Never 

□ b. 1-2 years 

□ c. 3-5 years 

□ d. 6-10 years 

□ e. More than 10 years 
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9. For how many years have academically focused before- or afterschool services been offered 
at this particular center (including before the beginning of the 21st Century Community 
Learning Centers grant)? 

(Mark ONE.) 

□ a. Never 

□ b. 1-2 years 

□ c. 3-5 years 

□ d. 6-10 years 

□ e. More than 10 years 

10. What policies regarding attendance does the center have for students? (Mark ONE.) 

□ a. Attendance is not required; students can drop in and out. 

□ b. We require students to attend at least 1 day per week. 

□ c. We require students to attend 2-3 days per week. 

□ d. We require students to attend every day we offer services. 

□ e. Other (specify): 

1 1 . What consequences are there for students who do not meet the general attendance 
requirements of the center? (Mark ALL that apply.) 

□ a. Not applicable; we do not require attendance. 

□ b. A staff member speaks to the student about an absence after each absence. 

□ c. A staff member speaks to the student about an absence after a certain number of 

absences. 

□ d. The student is restricted from participating in other activities at the center for a 

period of time. 

□ e. The student receives “points” or some other kind of penalty that restricts privileges 

at the center. 

□ f. A staff member notifies the student’s parents or caregivers. 

□ g. A staff member notifies the student’s school. 

□ h. Other (specify): 

12. Does the program charge a fee for students to attend, and if so, how much is the fee 
(calculated on a monthly basis)? (Mark ONE.) 

□ a. The program does not charge a fee 

□ b. Under $10 per month 

□ c. Between $10 and $25 per month 

□ d. Between $26 and $50 per month 

□ e. More than $50 per month 
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Tracking Outcomes 



13. What information is available to the center from school or district officials on student 
performance? (Mark ALL that apply in each row.) 

Reading/ Other 

language arts Mathematics subjects 



a. Test results from the NCLB -required 
state assessment for the whole school 


□ 


□ 


□ 


b. NCLB state assessment test results for 
the individual students at your center 


□ 


□ 


□ 


c. Information about students’ grades 


□ 


□ 


□ 


d. Individual information from school- 
administered diagnostic tests 


□ 


□ 


□ 


e. Data from students’ academic portfolios 


□ 


□ 


□ 


f. Quarterly performance test results 


□ 


□ 


□ 


g. End-of-course test results 


□ 


□ 


□ 


h. We do not receive information on student 
performance. 


□ 


□ 


□ 



14. Do you assess students’ academic growth in the program as part of your program 
activities? 

□ Yes □ No If no, skip to Question 17. 
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15. How often does your center use the following information to assess students’ academic 
growth in the program, either for making adjustments to the program or for evaluating the 
program’s success? (Mark ONE in each row.) 







Not 

used 


Use 1-2 
times a 
year 


Use 1-2 
times 
per 
month 


Use 

nearly 

every 

week 


Use almost 
daily 


a. 


Informal staff observations 


□ 


□ 


□ 


□ 


□ 


b. 


Written reports from teachers 


□ 


□ 


□ 


□ 


□ 


c. 


Verbal reports from teachers 


□ 


□ 


□ 


□ 


□ 


d. 


Results from tests administered by 
center staff 


□ 


□ 


□ 


□ 


□ 


e. 


Results from tests administered at 
students’ schools 


□ 


□ 


□ 


□ 


□ 


f. 


Student self-report or self-assessment 


□ 


□ 


□ 


□ 


□ 


g- 


Informal discussions with family 
members 


□ 


□ 


□ 


□ 


□ 


h. 


Other (specifv): 


□ 


□ 


□ 


□ 


□ 



16. How does your center use the results of information from student assessments of academic 
growth? (Mark ALL that apply.) 

□ a. To identify needs for new program offerings. 

□ b. To make improvements to practices within existing program offerings. 

□ c. To assign students to particular activities. 

□ d. To assign students to particular staff who are qualified to help students in identified 

academic areas of need. 

□ e. To assign students to particular groups or to older youth in the program. 

□ f. To evaluate the success of the program. 

□ g- We do not use these data in our programming. 

17. How do you use attendance data in your center? (Mark ALL that apply.) 

□ a. We keep the attendance data on file for the grant director or evaluator to use. 

□ b. We track individual student attendance to ensure that they come regularly. 

□ c. We use data on attendance to adjust program offerings. 

□ d. Other (specify): 
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18. What kind of feedback do students receive about the progress they are making or their 
accomplishments at the center? (Mark ALL that apply.) 

□ a. Verbal feedback from center staff on work assigned by the school 

□ b. Written feedback from center staff on work assigned by the school 

□ c. Verbal feedback from center staff on work they assign at the center 

□ d. Written feedback from staff on work they assign at the center 

□ e. Verbal feedback from center staff on behavior 

□ f. Written feedback from center staff on student behavior 

□ g- “Points” or rewards for program accomplishments 

□ h. Certificates or awards for accomplishments 

19. What kind of feedback does the center send to parents about the progress their children are 
making in the before- or afterschool program? (Mark ALL that apply.) 

□ a. Feedback from center staff on work assigned by the school 

□ b. Feedback from center staff on work assigned by the center 

□ c. Feedback from center staff on behavior 

□ d. “Points” or rewards for program accomplishments 

□ e. Certificates or awards for accomplishments 

□ f. We do not send written or verbal feedback to parents. 



How often do center staff communicate with parents about their students’ needs 
progress in each of the ways listed below? (Mark ONE in each row.) 


and 




Never or 
hardly 
ever 


Once or 
twice a 
term 


Once or 
twice a 
month 


Almost 

every 

week 


a. Talk with parents when they pick up 
their child about the child’s 
difficulties and/or progress 


□ 


□ 


□ 


□ 


b. Contact parents by phone or e-mail 


□ 


□ 


□ 


□ 


c. Send written reports of progress at the 
before- or afterschool program home 
with students 


□ 


□ 


□ 


□ 


d. Hold center staff-parent conferences 


□ 


□ 


□ 


□ 


e. Involve parents in planning new 
programming to match students’ 
needs 


□ 


□ 


□ 


□ 


f. Hold an event in which parents can 
come and see student academic work 


□ 


□ 


□ 


□ 
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student Academic Support Activities 

Reading Support Activities 

Academic support in reading refers to structured activities designed to build students’ literacy 
skills. Academic support activities may include scheduled time for independent student reading, 
writing, and literacy enrichment activities, but NOT homework assistance. Note: Depending 
upon the age groups and needs of students you serve, some of these reading tasks may not be 
appropriate for your students. For these questions, please consult with the staff memher(s) who 
lead reading support activities at your center. 

21. In what proportion of your reading activities are students asked to do the following? 

(Mark ONE in each row.) 

None 

or 





almost 

none 


Some 

activities 


Most 

activities 


All 

activities 


a. Read aloud 


□ 


□ 


□ 


□ 


b. Talk with peers about what they have read 


□ 


□ 


□ 


□ 


c. Write about something they have read 


□ 


□ 


□ 


□ 


d. Work in a reading workbook or on a worksheet 


□ 


□ 


□ 


□ 


e. Read books students have chosen themselves 


□ 


□ 


□ 


□ 


f. Read books selected by the leader 


□ 


□ 


□ 


□ 


g. Do a group activity or project about what they have 
read 


□ 


□ 


□ 


□ 


h. Discuss different interpretations of what they have read 


□ 


□ 


□ 


□ 


i. Explain or support their understanding of what they 
have read 


□ 


□ 


□ 


□ 


j. Watch videos/DVDs or listen to music 


□ 


□ 


□ 


□ 


k. Help students understand new words 


□ 


□ 


□ 


□ 


1. Talk or write about answers to questions about 
something they have read 


□ 


□ 


□ 


□ 


m. Make predictions about something they are reading 
before reading or as part of reading 


□ 


□ 


□ 


□ 


n. Make generalizations and draw inferences about 
something they have read 


□ 


□ 


□ 


□ 


o. Describe the style or structure of a text they have read 


□ 


□ 


□ 


□ 
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22. In what proportion of activities at your center aimed at promoting reading or literacy skills 
do instructors emphasize to the following reading processes? (Mark ONE in each row.) 



None or 

almost Some Most All 

none activities activities activities 




Mathematics Support Activities 

Academic support in mathematics refers to structured activities dedicated to increasing students’ 
mathematics skills. Academic support activities may include applied enrichment, skill building, 
mathematics games, and scheduled time for tutoring, but NOT homework assistance. Note: 
Depending upon the age groups and needs of students you serve, some of these mathematics 
tasks may not be appropriate for your students. For these questions, please consult with the 
staff memher(s) who lead mathematics support activities at your center. 

23. In what proportion of your mathematics activities are students asked to do the following? 
(Mark ONE in each row.) 

None or 





almost 

none 


Some 

activities 


Most 

activities 


All 

activities 


a. Explain the reasoning behind how they 
solved a problem 


□ 


□ 


□ 


□ 


b. Make charts, tables, or graphs 


□ 


□ 


□ 


□ 


c. Solve problems in which students have to 
figure out what method to use to solve them 


□ 


□ 


□ 


□ 


d. Solve problems that have many possible 
correct answers 


□ 


□ 


□ 


□ 


e. Use computers to practice math 


□ 


□ 


□ 


□ 


f. Write mathematical equations to solve a 
problem 


□ 


□ 


□ 


□ 


g. Practice basic math facts (e.g., addition, 
subtraction, multiplication, and division) 


□ 


□ 


□ 


□ 


h. Use graphing calculators to solve problems 


□ 


□ 


□ 


□ 
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24. In what proportion of activities at your center aimed at promoting mathematics skills do 
instructors give emphasis to the following? (Mark ONE in each row.) 







None or 
almost 
none 


Some 

activities 


Most 

activities 


All 

activities 


a. 


Number sense 


□ 


□ 


□ 


□ 


b. 


Operations (e.g., addition, subtraction, 
multiplication, division) with whole 
numbers 


□ 


□ 


□ 


□ 


c. 


Operations with decimals or fractions 


□ 


□ 


□ 


□ 


d. 


Measurement 


□ 


□ 


□ 


□ 


e. 


Algebraic concepts or algebra (e.g., 
solving equations) 


□ 


□ 


□ 


□ 


f. 


Geometric concepts or geometry (e.g., 
area and perimeter of shapes) 


□ 


□ 


□ 


□ 


g- 


Data analysis (e.g., mean, median, 
mode) 


□ 


□ 


□ 


□ 



Use of Technology Resources 



25. How many Internet-connected computers (desktop or laptop) are available for student use 

at this center? 

□ a. None 

□ b. Less than 5 

□ c. 6-10 

□ d. More than 10 

26. On average, how often have students at your center used the following kinds of software or 
Internet applications in the past programming year? (Mark ONE in each row.) 







Never or 


Once or 


Once or 


Almost 






hardly 


twice a 


twice a 


every 






ever 


month 


week 


day 


a. 


Internet search tools (e.g., Google, Yahoo!) 


□ 


□ 


□ 


□ 


b. 


Word processing, spreadsheet, or 
presentation software (e.g., Microsoft 
Word, Excel, PowerPoint) 


□ 


□ 


□ 


□ 


c. 


Drill and practice software for mathematics 
(e.g.. Achieve Now, SuccessMaker) 


□ 


□ 


□ 


□ 


d. 


Drill and practice software for reading (e.g.. 
Destination Reading, READ 180) 


□ 


□ 


□ 


□ 
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e. Simulation or modeling software or applets 
that run over the Internet 


□ 


□ 


□ 


□ 


f. Photo or video editing software (e.g., 
Photoshop, iMovie) 


□ 


□ 


□ 


□ 


g. Web page design tools (e.g., Dreamweaver) 


□ 


□ 


□ 


□ 



staffing 

staff Pay and Benefits 

27. How many of the following types of staff does your center employ? (Mark ALL that 
apply.) 

□ Certified teachers: 

□ School paraprofessional staff: 

□ Other professionals (e.g., youth-development specialists): 

□ Other paraprofessionals (e.g., clerks, administrative assistants): _ 

□ Unpaid volunteers: 

28. In addition to wages, what benefits are offered to full- and part-time staff? 

(Mark ALL that apply in each row.) 





Full-time 

staff 


Part-time 

staff 


Not 

offered 


Don’t 

know 


a. Paid time off for vacation and sick 
leave 


□ 


□ 


□ 


□ 


b. Health insurance 


□ 


□ 


□ 


□ 


c. Retirement savings plan 


□ 


□ 


□ 


□ 


d. Paid attendance at conferences 


□ 


□ 


□ 


□ 


e. Paid training or professional 
development 


□ 


□ 


□ 


□ 


f. Tuition reimbursement 


□ 


□ 


□ 


□ 
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staff Retention 



29. Please list the number of staff, aside from you, who have worked at the center (at least 5 
hours per week) for the following durations. (Write a number on each line or check “Don’t 
know.”) 



Number of staff Don’t know 



a. 


Up to 1 school term 


□ 


b. 


More than 1 school term up to 1 vear 


□ 


c. 


More than 1 vear up to 2 vears 


□ 


d. 


More than 2 vears up to 5 vears 


□ 


e. 


More than 5 vears 


□ 



30a. How many staff who work at least 5 hours per week have left the program since the start of 
the 2005-06 school year? 

staff 

30b. What are the primary reasons why staff leave the program? (Mark ALL that apply.) 

□ a. We have not had staff leave our program. 

□ b. Low pay 

□ c. Lack of benefits 

□ d. Inability to offer a full-time position 

□ e. Completion/graduation from a program (e.g., Americorps) or from school 

□ f. Other (specify: ) 

□ g. Don’t know 
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Overall Program Challenges 

3 1 . How much of a challenge to implementing high-quality programming at your center is each 
of the following? (Mark ONE in each row.) 





Not a 
challenge 


A minor 
challenge 


A major 
challenge 


a. The school would like our program to be more academically 
focused. 


□ 


□ 


□ 


b. The space available for center programs is inadequate, 
inappropriate, or unsafe. 


□ 


□ 


□ 


c. We cannot find staff with expertise in teaching the academic 
subjects we offer. 


□ 


□ 


□ 


d. We cannot find volunteers with time and expertise to support 
academic activities at our center. 


□ 


□ 


□ 


e. We cannot afford to offer competitive salaries to staff who 








are qualified to provide supplementary academic instruction 
at our center. 


□ 


□ 


□ 


f. We cannot afford to offer potential staff enough hours of 
paid employment. 


□ 


□ 


□ 


g. Staff do not come to work on a reliable schedule. 


□ 


□ 


□ 


h. Volunteers are not available on a reliable schedule. 


□ 


□ 


□ 


i. Staff do not have the skills to help English-language learners 
with their academic development. 


□ 


□ 


□ 


j. There are limited professional development opportunities 
for staff. 


□ 


□ 


□ 


k. We have inadequate instructional materials or programming 


□ 


□ 


□ 


ideas. 


1. We receive insufficient information about how to help center 


□ 


□ 


□ 


participants with the school day curriculum. 


m. There is insufficient information about the academic needs 


□ 


□ 


□ 


of students at our center. 


n. Students face transportation barriers in getting to the center. 


□ 


□ 


□ 


o. Students do not attend the center regularly enough to make 


□ 


□ 


□ 


academic improvements. 


p. Students are not interested in coming to the program. 


□ 


□ 


□ 


q. Students drop out because they lose interest in doing 
academic work after school. 


□ 


□ 


□ 


r. The school’s staff do not respond to center requests to 
coordinate services or resources for participants. 


□ 


□ 


□ 


s. Families are not sufficiently involved in monitoring the 
academic or behavior progress their children make at the 


□ 


□ 


□ 


center. 








t. There is too much competition for students’ time from other 


□ 


□ 


□ 


activities, such as jobs or sports. 
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Thank you for your participation. 
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21st Century Community Learning Centers Program 

Quality Study 



Center Director Focused Survey 



Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless such collection displays a valid 0MB control number. The valid 0MB control number for this information collection is 
1875-0239. The time required to complete this information collection is estimated to average 1 .5 hours per response, including the time to 
review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have 
any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of 
Education, Washington, D.C. 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, 
write directly to: Jessica Hausman, Policy and Program Studies Service, U.S. Department of Education, 400 Maryland Ave, S.W., Room 
6W227, Washington, D.C. 20202. 
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21st Century Community Learning Centers 
Program Quaiity Study 

Center Director FOCUSED Survey 
March 2006 



Note to Survey Recipient: 

This survey should he completed primarily hy the director of the 21st Century Community 
Learning Center. Most of the questions can he easily answered hy or are specific to the center 
director. For questions 15-20, we encourage you to discuss responses with the grant director (if 
that is not you). In addition, we encourage you to confer with the three previously identified 
staff members for questions 25-45 of this survey and with any other appropriate staff members 
for other items that may be difficult for you to answer. Answer all questions in the survey only 
for the center indicated in the cover sheet to the survey. 



Topics addressed in the survey include: 

Student Participation 
Recruitment 
Attendance 
Targeting Services 

Tracking of Student Scores on State Tests 

Coordination with Supplemental Educational Services Providers 

Staffing and Training 

Center Director’s Background and Experience 
Roles, Backgrounds, and Experiences of Specific Staff Members 
Organization of Instruction in Reading and Mathematics Activities 



Please record the names and titles of all respondents in the spaces below. 



Name(s)/Title(s) of the person(s) responding to 
this survey: 


Contact information (telephone or e-mail): 


Primary Respondent 


Telephone 

E-mail 


Name: 

Title: 


Telephone 




E-mail 


Respondent for Items 25-45 

Name: 

Title: 
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Student Participation 

Recruitment 



1 . Which of the following groups of students does your center seek to serve? 

(Mark ALL that apply.) 

□ a. We have open enrollment for all interested students. 

□ c. Students who scored “below proficient” on state assessments. 

□ d. Students identified by the school as needing special assistance in reading and/or 

math. 

□ e. Students who are English-language learners. 

□ f. Students who receive free or reduced-priced meals. 

□ g- Students who are recommended by teachers or counselors at the feeder schools. 

□ h. Students with siblings already attending. 

□ i. Other (specify): 

2. Which, if any, of the groups listed in question 1 have priority for the center’s services? 
(Select THREE groups from question 1 and indicate them by marking the corresponding 
letter in the rows below. List the top priority group first, followed by the second and third 
priority groups. Mark ONE in each row.) 







a 


h 


c 


d 


e 


f 


g 


h 


a. 


Top priority 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


b. 


Second priority 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


c. 


Third priority 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



3. How many total students are currently enrolled at this center? 



4. What is the maximum number of students the center can serve? 
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5. Which of the following strategies do you use to engage and retain students at the center? 

(Mark ALL that apply.) 

□ a. Offering targeted academic instruction in reading and mathematics. 

□ b. Offering academic enrichment activities, such as free reading, hands-on projects in 

content areas (e.g., science, history, geography projects), or field trips. 

□ c. Providing individualized or small group tutoring. 

□ d. Allowing students to choose the activities they participate in. 

□ e. Providing both space and time at the center for recreation, snacks, and talking with 

friends. 

□ f. Providing a full meal after school. 

□ g. Providing family literacy assistance. 

□ h. Providing transportation to the center. 

□ i. Offering incentives (e.g., field trips, movie tickets) for participation. 

Attendance 

6. How does this center track attendance? (Mark ONE.) 

□ a. Pen and paper records. 

□ b. Using a database management program on a single computer. 

□ c. Using a Web-based data system for recording and reporting data on a computerized 

network. 

□ d. Using student IDs with barcodes and an electronic scanner linked to computers. 

□ e. Other (specify): 

7. What kinds of attendance reports can your center generate easily (within 1-2 hours time)? 

(Mark ALL that apply.) 

□ a. Total number of students who attended the center in the past week. 

□ b. Total number of students who attended the center in the past school year. 

□ c. Number of times each student attends, recorded by the week. 

□ d. Number of times each student attends, recorded by the day. 

□ e. Number of special education students who have attended the program since the 

school year began. 

8. If you selected “d” above in Question 7, answer this question: Last year (2004-05), how 
many students attended this center for the following amounts of time? (Write a number on 
each line below. Write “0” if there are no students in a category.) 

□ a. This information is not available in this format for our center. 



Appendix D: Survey Instruments 



116 




Middle 

school students Secondary 
(typically school students 
grades 6-8) (grades 9-12) 

b. Attended fewer than 30 days 

c. Attended 30 to 59 days 

d. Attended 60 to 89 days 

e. Attended 90 to 119 days 

f. Attended 120 or more days 



Elementary 
school students 
(typically 

Number of students who: grades K-5) 



9. For each of the activities below, mark the box that most accurately corresponds to your 
center’s policies regarding student attendance for the activities indicated. 

(Mark ONE in each row.) 

□ a. This information is not available in this format for our center. 



Activity 

not 

offered 



Required 
for low- 

Optional for achieving 
all students students 



Required 
of ALL 
students 



b. Homework help in a group 
setting 


□ 


□ 


□ 


□ 


c. Tutoring (one-to-one or peer) 


□ 


□ 


□ 


□ 


d. Academic enrichment activities 
in which reading instruction or 
practice takes place (not 
tutoring) 


□ 


□ 


□ 


□ 


e. Academic enrichment activities 
in which mathematics 
instruction or practice takes 
place (not tutoring) 


□ 


□ 


□ 


□ 
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Tracking of Student Scores on State Tests 

10. Has your center received data from participants’ schools that tells how they performed on 
the state assessment test used for NCLB in 2004-05? (Mark ONE.) 

□ a. I don’t know. Skip to Question 13. 

□ b. No. Skip to Question 13. 

□ c. Yes. 

11. Do you have knowledge of the level of proficiency of individual students on these tests? 

(Mark ONE.) 

□ a. No. Skip to Question 13. 

□ b. Yes. 



12. Approximately what percent of participants at your center scored below proficient on the 
most recent assessment that the state administers for NCLB accountability? 

□ a. I don’t know. 

□ b. Percent scoring below proficient in READING: 

□ c. Percent scoring below proficient in MATHEMATICS: 

13. Have you or has the school administered any mid-year assessments that could help predict 
whether students scoring “below proficient” are likely to score “proficient” or higher on the 
2005-06 state tests used for NCEB? (Mark ONE.) 

□ a. I don’t know. Skip to Question 15. 

□ b. No. Skip to Question 15. 

□ c. Yes. 

14. Approximately what percent of participants enrolled in your center are likely to move from 
scoring “below proficient” to “proficient” on 2005-06 state tests, compared with last year? 

□ a. I don’t know. 

□ b. Percent likely to move from “below proficient” to “proficient” in READING: 

□ c. Percent likely to move from “below proficient” to “proficient” in 

MATHEMATICS: 
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Coordination with Suppiementai Services Providers 



15. This year, in 2005-06, is your center a state-approved provider of supplemental educational 
services (SES)? (Mark ONE.) 

□ a. I don’t know. Skip to Question 17. 

□ b. No. Skip to Question 17. 

□ c. Yes. 

16. If your center is a state-approved provider of supplemental educational services under 
NCLB, how many SES students does your center serve THIS SEMESTER? 

□ a. I don’t know. 

□ b. Our center serves students in READING under SES this semester. 

□ c. Our center serves students in MATHEMATICS under SES this 

semester. 



17. This year, in 2005-06, does your center collaborate with other supplemental education 
services providers to offer services to participants at your center? (Mark ONE.) 

□ a. No. Skip to Question 20. 

□ b. Yes. 

18. With how many different supplemental service providers does your center collaborate? 



19. How do you coordinate services with those SES providers? (Mark ALL that apply.) 

□ a. The SES provider(s) help us decide the focus of our programs. 

□ b. We coordinate our schedule with the SES provider(s). 

□ c. We coordinate academic support activities such as tutoring for individual students. 

□ d. We help each other recruit students. 

□ e. We work together to communicate with students’ schools. 

□ f. Other (specify): 

20. Do students who attend activities at your center receive SES as defined by NCEB from a 
provider other than your center? (Mark ONE.) 

□ a. I don’t know. 

□ b. No. 

□ c. Yes. 
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Staffing and Training 

Center Director’s Background and Experience 

21. What is the highest level of education you have completed? (Mark ONE.) 

□ a. Less than high school 

□ b. High school or GED 

□ c. Some college, other classes/training not related to a degree 

□ d. Completed 2-year college degree. Major: 

□ e. Completed 4-year college degree. Major: 

□ f. Some graduate work. Discipline: 

□ g. Master’s degree. Discipline: 

□ h. Ph.D. Discipline: 

22. Are you certified to teach in school? (Mark ONE.) 

□ a. No. 

□ b. I have a provisional or emergency teaching credential. 

□ c. I have a single- subject certification. 

□ d. I have a multisubject certification. 
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23. What types of experience did you have before you assumed your current position at this 
21st Century Center? How long were you in these positions? (Mark ONE in each row.) 







Less 






More 






than a 


1-3 


4-6 


than 6 


Your previous experience 


None 


year 


years 


years 


years 


a. Classroom teacher 


□ 


□ 


□ 


□ 


□ 


b. Classroom aide/teaching assistant 
(paraprofes sional) 


□ 


□ 


□ 


□ 


□ 


c. Instructional specialist in reading or 
mathematics 


□ 


□ 


□ 


□ 


□ 


d. Instructional specialist in arts or 
recreation 


□ 


□ 


□ 


□ 


□ 


e. Pupil support staff (e.g., school 
counselor, social worker, 


□ 


□ 


□ 


□ 


□ 


psychologist) 












f. School administrator 


□ 


□ 


□ 


□ 


□ 


g. Administrator at a child/youth center 


□ 


□ 


□ 


□ 


□ 


or at a park or recreation center 


h. Administrator in a social services 












organization (e.g., health services, 
welfare office) 


□ 


□ 


□ 


□ 


□ 


i. Recreation, youth, or childcare 
worker 


□ 


□ 


□ 


□ 


□ 


j. Social services or health services 
provider 


□ 


□ 


□ 


□ 


□ 


k. Camp counselor/leader 


□ 


□ 


□ 


□ 


□ 


1. Other (snecifv): 


□ 


□ 


□ 


□ 


□ 
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24. What roles and responsibilities do you have at the center? (Mark ALL that apply.) 

□ a. Supervising paid staff 

□ b. Helping develop or train staff 

□ c. Coaching and mentoring staff 

□ d. Coordinating with school day staff or activities 

□ e. Securing and maintaining space for activities 

□ f . Managing center budgets 

□ g. Fundraising or grant writing 

□ h. Community outreach and partnership development 

□ i. Internal evaluation of programs 

□ j- Instructor for reading activities 

□ k. Instructor for mathematics activities 

□ 1. Instructor for other enrichment activities 

□ m. Tutor for individual students 

□ n. Counselor or social worker 

Staff Roles, Backgrounds, and Experience 

Please answer each of the following sets of questions for the three (3) staff members that we 
identified from the roster you submitted to us. For each, first indicate the staff member’s role at 
the center, and then answer the questions about his/her background and professional 
development experiences. If your center does not have three staff members, answer questions 
about as many staff members as we identified from the roster you submitted. If you do not have 
the information for a staff member, a researcher can contact that staff member directly by 
telephone or e-mail to set up a time to gather that information. 



25. STAFF MEMBER 1: What roles does this staff member play at the center? (Mark ALL 
that apply.) 

□ a. Provides homework assistance 

□ b. Provides literacy- or reading-related services or instruction 

□ c. Provides mathematics-related services or instruction 

□ d. Leads activities in (specify): 

□ e. Other role (specify): 
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26. Approximately how many hours each week is this staff member paid to work at this center, 
including any paid time for training and staff development? (Mark ONE.) 

□ a. Full -time (35 hours or more per week) 

□ b. Half-time (20-34 hours per week) 

□ c. Part-time (10-19 hours per week) 

□ d. Occasionally (less than 10 hours per week) 

27. How long has this staff member been in this position? (Mark ONE.) 

□ a. Up to 1 school term 

□ b. More than 1 school term up to 1 year 

□ c. More than 1 year up to 2 years 

□ d. More than 2 years up to 5 years 

□ e. More than 5 years 

28. What is the highest level of education this staff member has completed? (Mark ONE.) 

□ a. Less than high school 

□ b. High school or GED 

□ c. Some college, other classes/training not related to a degree 

□ d. Completed 2-year college degree. Major: 

□ e. Completed 4-year college degree. Major: 

□ f. Some graduate work. Discipline: 

□ g. Master’s degree. Discipline: 

□ h. Ph.D. Discipline: 

29. Is this staff member certified to teach in public school? (Mark ONE.) 

□ a. No. 

□ b. He/she has a provisional or emergency teaching credential. 

□ c. He/she has a single-subject certification. 

□ d. He/she has a multisubject certification. 
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30. What types of experience did this staff member have before joining the staff of the center? 
How long was the staff member in each position? (Mark ONE in each row.) 







Less 






More 






than a 


1-3 


4-6 


than 6 


Previous experience 


None 


year 


years 


years 


years 


a. Paid recreation, youth, or 












childcare worker in other before- 
or afterschool program 


□ 


□ 


□ 


□ 


□ 


b. Volunteer youth worker in other 
before or afterschool program 


□ 


□ 


□ 


□ 


□ 


c. Noninstructional paraprofessional 
(e.g., security, cafeteria, social 


□ 


□ 


□ 


□ 


□ 


services worker) 












d. Paraprofessional classroom aide 


□ 


□ 


□ 


□ 


□ 


e. Instructional specialist in reading or 
mathematics 


□ 


□ 


□ 


□ 


□ 


f. Instructional specialist in arts or 
recreation 


□ 


□ 


□ 


□ 


□ 


g. Certified classroom teacher 


□ 


□ 


□ 


□ 


□ 


h. Administrator at a child/youth 












center, a park or recreation center, or 
social services organization 


□ 


□ 


□ 


□ 


□ 


i. Administrator in a social services 












organization (e.g., health services, 
welfare office) 


□ 


□ 


□ 


□ 


□ 


j. Camp counselor/leader 


□ 


□ 


□ 


□ 


□ 


k. Other (specifv): 


□ 


□ 


□ 


□ 


□ 



31. STAFF MEMBER 2: What roles does this staff member play at the center? (Mark ALL 
that apply.) 

□ a. Provides homework assistance 

□ b. Provides literacy- or reading-related services or instruction 

□ c. Provides mathematics-related services or instruction 

□ d. Leads activities in (specify): 

□ e. Other role (specify): 
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32. Approximately how many hours each week is this staff member paid to work at this center, 
including any paid time for training and staff development? (Mark ONE.) 

□ a. Full -time (35 hours or more per week) 

□ b. Half-time (20-34 hours per week) 

□ c. Part-time (10-19 hours per week) 

□ d. Occasionally (less than 10 hours per week) 

33. How long has this staff member been in this position? (Mark ONE.) 

□ a. Up to 1 school term 

□ b. More than 1 school term up to 1 year 

□ c. More than 1 year up to 2 years 

□ d. More than 2 years up to 5 years 

□ e. More than 5 years 

34. What is the highest level of education this staff member has completed? (Mark ONE.) 

□ a. Less than high school 

□ b. High school or GED 

□ c. Some college, other classes/training not related to a degree 

□ d. Completed 2-year college degree. Major: 

□ e. Completed 4-year college degree. Major: 

□ f. Some graduate work. Discipline: 

□ g. Master’s degree. Discipline: 

□ h. Ph.D. Discipline: 

35. Is this staff member certified to teach in public school? (Mark ONE.) 

□ a. No. 

□ b. He/she has a provisional or emergency teaching credential. 

□ c. He/she has a single-subject certification. 

□ d. He/she has a multisubject certification. 
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36. What types of experience did this staff member have before joining the staff of the center? 
How long was the staff member in each position? (Mark ONE in each row.) 







Less 






More 






than a 


1-3 


4-6 


than 6 


Previous experience 


None 


year 


years 


years 


years 


a. Paid recreation, youth, or childcare 












worker in other before- or 
afterschool program 


□ 


□ 


□ 


□ 


□ 


b. Volunteer youth worker in other 
before or afterschool program 


□ 


□ 


□ 


□ 


□ 


c. Noninstructional paraprofessional 
(e.g., security, cafeteria, social 


□ 


□ 


□ 


□ 


□ 


services worker) 












d. Paraprofessional classroom aide 


□ 


□ 


□ 


□ 


□ 


e. Instructional specialist in reading or 
mathematics 


□ 


□ 


□ 


□ 


□ 


f. Instructional specialist in arts or 
recreation 


□ 


□ 


□ 


□ 


□ 


g. Certified classroom teacher 


□ 


□ 


□ 


□ 


□ 


h. Administrator at a child/youth 












center, a park or recreation center, or 
social services organization 


□ 


□ 


□ 


□ 


□ 


i. Administrator in a social services 












organization (e.g., health services, 
welfare office) 


□ 


□ 


□ 


□ 


□ 


j. Camp counselor/leader 


□ 


□ 


□ 


□ 


□ 


k. Other (specifv): 


□ 


□ 


□ 


□ 


□ 



37. STAFF MEMBER 3: What roles does this staff member play at the center? (Mark ALL 
that apply.) 

□ a. Provides homework assistance 

□ b. Provides literacy- or reading-related services or instruction 

□ c. Provides mathematics-related services or instruction 

□ d. Leads activities in (specify): 

□ e. Other role (specify): 
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38. Approximately how many hours each week is this staff member paid to work at this center, 
including any paid time for training and staff development? (Mark ONE.) 

□ a. Full -time (35 hours or more per week) 

□ b. Half-time (20-34 hours per week) 

□ c. Part-time (10-19 hours per week) 

□ d. Occasionally (less than 10 hours per week) 

39. How long has this staff member been in this position? (Mark ONE.) 

□ a. Up to 1 school term 

□ b. More than 1 school term up to 1 year 

□ c. More than 1 year up to 2 years 

□ d. More than 2 years up to 5 years 

□ e. More than 5 years 

40. What is the highest level of education this staff member has completed? (Mark ONE.) 

□ a. Less than high school 

□ b. High school or GED 

□ c. Some college, other classes/training not related to a degree 

□ d. Completed 2-year college degree. Major: 

□ e. Completed 4-year college degree. Major: 

□ f. Some graduate work. Discipline: 

□ g. Master’s degree. Discipline: 

□ h. Ph.D. Discipline: 

41. Is this staff member certified to teach in public school? (Mark ONE.) 

□ a. No. 

□ b. He/she has a provisional or emergency teaching credential. 

□ c. He/she has a single-subject certification. 

□ d. He/she has a multisubject certification. 
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42. What types of experience did this staff member have before joining the staff of the center? 
How long was the staff member in each position? (Mark ONE in each row.) 

Less More 

than a 1-3 4-6 than 6 

Previous experience None year years years years 

a. Paid recreation, youth, or childcare 





worker in other before or afterschool 
program 


□ 


□ 


□ 


□ 


□ 


b. 


Volunteer youth worker in other 
before- or afterschool program 


□ 


□ 


□ 


□ 


□ 


c. 


Noninstructional paraprofessional 
(e.g., security, cafeteria, social 
services worker) 


□ 


□ 


□ 


□ 


□ 


d. 


Paraprofessional classroom aide 


□ 


□ 


□ 


□ 


□ 


e. 


Instructional specialist in reading or 
mathematics 


□ 


□ 


□ 


□ 


□ 


f. 


Instructional specialist in arts or 
recreation 


□ 


□ 


□ 


□ 


□ 


g- 


Certified classroom teacher 


□ 


□ 


□ 


□ 


□ 


h. 


Administrator at a child/youth 
center, a park or recreation center, or 
social services organization 


□ 


□ 


□ 


□ 


□ 


i. 


Administrator in a social services 
organization (e.g., health services, 
welfare office) 


□ 


□ 


□ 


□ 


□ 


j- 


Camp counselor/leader 


□ 


□ 


□ 


□ 


□ 


k. 


Other (specify): 


□ 


□ 


□ 


□ 


□ 
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Organization of Instruction in Reading and Mathematics Activities 



Please describe one example of a READING activity at your center that you believe is 
particularly effective in reaching youth. Be sure to describe the content taught, the roles and 
responsibilities of staff, how students are grouped, student-to-adult ratio, and at least one lesson 
or activity that would help an outsider understand what it would be like to participate in the 
activity as a student. 



Please describe one example of a MATHEMATICS activity at your center that you believe is 
particularly effective in reaching youth. Be sure to describe the content taught, the roles and 
responsibilities of staff, how students are grouped, student-to-adult ratio, and at least one lesson 
or activity that would help an outsider understand what it would be like to participate in the 
activity as a student. 



Thank you for your participation. 
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Appendix E 

Supplementary Tables for the Surveys 




Background for This Appendix 

This appendix presents exhibits with descriptive and inferential statistics that have been used 
as the source of data for the exhibits presented in the main text of the 21st Century Community 
Learning Centers Program Quality study. Evaluation team analysts constructed the exhibits from 
responses to both the national survey and the focused telephone survey. 

Many of these exhibits compare the responses of centers in different settings (school-based 
versus nonschool-based) and different levels of students served (elementary, middle and high 
school). The overall results are presented only for items for which there were no statistically 
significant differences between center types or among grade levels. When such statistically 
significant differences did occur, in comparing results for center types and levels with 
statistically significant differences (at the p < 0.05 level), separate exhibits were constructed. 
Statistically significant findings have been marked with an asterisk (*). 

Interpreting differences among levels, however, was complicated because many centers 
served students at multiple levels. For example, some sites served both elementary and middle 
school students. A statistical comparison that did not take this fact into account would have been 
likely to lead to fewer observed differences among sites, and it would also have violated the 
assumption that observations were independent from one another. Therefore, when comparing 
levels, the team addressed this problem by statistically comparing sites that served elementary 
school children against all sites that did not serve children of that age range. Analysts similarly 
compared sites that did and did not serve middle school students and sites that did and did not 
serve high school students. An asterisk appearing in the elementary school column of an exhibit 
indicates that the responses of centers serving elementary school children were statistically 
significantly different from those centers that did not serve elementary school children. 




Exhibit E-1 

Centers Serving Elementary School Students That Reported Participation in Activities by 
a Typical Student, by Type of Activity and Amount of Time of Engagement per Week 



Activities 


Not 

offered 


Less 
than 2 
hours 




More 
than 4 
hours 


Homework help in a group setting 


3% 
(0.9%) 
n= 10 




36% 
(2.8%) 
n= 134 


36% 
(2.7%) 
n= 154 


An arts or recreation enrichment activity (e.g., 
sports, outdoor games, crafts, theater, music) 


1% 
(0.4%) 
n = 3 


28% 
(2.6%) 
n= 115 


38% 
(2.8%) 
n= 138 


33% 
(2.7%) 
n= 133 


An academic enrichment activity in which reading 
instruction or practice takes place (not tutoring) 


1% 

(0.6%) 
n = 3 


28% 
(2.6%) 
n= 116 


47% 
(2.9%) 
n= 173 


25% 
(2.5%) 
n = 92 


An academic enrichment activity in which 
mathematics instruction or practice takes place 
(not tutoring) 


4% 
(1 .0%) 
n= 14 


36% 
(2.8%) 
n= 149 


41% 
(2.8%) 
n= 148 


19% 
(2.3%) 
n = 77 


Tutoring (one-to-one or peer) 


11% 
(1 .9%) 
n= 39 


46% 
(2.9%) 
n= 179 


31% 
(2.7%) 
n= 118 


12% 
(1 .9%) 
n = 46 


A science activity, project or science-related field 
trip (e.g., to a local science museum) 


16% 
(2.1%) 
n = 68 


58% 
(2.9%) 
n = 223 


21% 

(2.4%) 

n=72 


6% 
(1 .3%) 
n = 21 


A service or other supplemental activity (e.g., 
counseling, drug and violence prevention, 
mentoring) 


23% 
(2.4%) 
n= 80 


60% 
(2.8%) 
n = 23^ 


15% 
(2.0%) 
n= 60 


3% 
(0.9%) 
n= 14 



Source: National Survey, Item 5. 

Numbers in parentheses are standard errors. 

Exhibit reads: Of those centers serving elementary school students, 24 percent of centers reported students 
participated in homework help in a group setting less than 2 hours a week. 
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Exhibit E-2 

Centers Serving Elementary School Students That Reported Reading Activities 
Emphases, by Reading Skill Emphasized and Inclusion Level 



Reading Skills 


None or 
almost 
none 


Some 

activities 


Most 

activities 


All 

activities 


Phonemic awareness 


8% 
(1 .5%) 
n = 32 


43% 
(2.9%) 
n= 168 


35% 
(2.8%) 
n= 132 


15% 
(2.1%) 
n = 53 


Phonics 


7% 
(1 .4%) 
n = 27 


41% 
(2.9%) 
n= 158 


38% 
(2.8%) 
n= 148 


14% 
(2.1%) 
n = 52 


Vocabulary 


3% 
(0.9%) 
n= 12 


28% 
(2.6%) 
n= 107 


44% 
(2.9%) 
n= 172 


25% 
(2.5%) 
n = 97 


Fluency 


6% 
(1 .3%) 
n = 25 


35% 
(2.8%) 
n= 139 


39% 
(2.8%) 
n= 152 


20% 
(2.4%) 
n = 72 


Comprehension 


3% 
(0.9%) 
n= 10 


24% 
(2.4%) 
n = 96 


43% 
(2.8%) 
n= 163 


30% 
(2.7%) 
n= 119 



Note; Percentages may not equal 100 percent due to rounding. 

Source; National Survey, Item 22. 

Numbers in parentheses are standard errors. 

Exhibit reads; Of those centers serving elementary school students, 43 percent or 168 of those centers reported 
students’ phonemic awareness was emphasized in some reading enrichment activities. 



Appendix E: Supplementary Tables for the Surveys 



135 




































Exhibit E-3 

Centers Serving Elementary School Students That Reported Student Participation in 
Instructional Strategies in Reading Activities, by Strategy and Inclusion Level 



Instructional Strategies 


None or 
almost 
none 


Some 

activities 


Most 

activities 


Aii 

activities 


Read aloud 


7% 

(1 .6%) 
n = 25 


57% 
(2.9%) 
n = 231 


30% 
(2.7%) 
n = 110 


6% 
(1 .3%) 
n = 20 


Talk with peers about what they have read 


2% 
(0.8%) 
n = 14 


62% 
(2.8%) 
n = 239 


29% 
(2.6%) 
n = 106 


7% 
(1 .5%) 
n = 24 


Write about something they have read 


8% 

(1 .6%) 
n = 30 


62% 
(2.9%) 
n = 240 


27% 
(2.6%) 
n = 97 


4% 
(1 .3%) 
n = 14 


Read books selected by the leader 


10% 
(1.7%) 
n = 46 


72% 
(2.6%) 
n = 267 


16% 
(2.1%) 
n = 63 


2% 
(0.8%) 
n = 8 


Do a group activity or project about what they 
have read 


10% 
(1.7%) 
n = 41 


60% 
(2.8%) 
n = 235 


26% 
(2.6%) 
n = 90 


5% 

(1 .2%) 
n = 19 


Discuss different interpretations of what they have 
read 


14% 
(2.0%) 
n = 54 


57% 
(2.9%) 
n = 230 


24% 
(2.5%) 
n = 83 


5% 

(1 .2%) 
n = 18 


Explain or support their understanding of what 
they have read 


7% 
(1 .4%) 
n = 32 


43% 
(2.8%) 
n = 173 


42% 
(2.8%) 
n = 152 


8% 

(1 .6%) 
n = 31 


Help other students understand new words 


7% 
(1 .3%) 
n = 33 


57% 
(2.9%) 
n = 220 


25% 
(2.5%) 
n = 89 


11% 
(1 .9%) 
n = 43 


Talk or write about answers to questions about 
something they have read 


6% 
(1 .3%) 
n = 29 


52% 
(2.9%) 
n = 204 


31% 
(2.7%) 
n = 111 


11% 
(1 .8%) 
n = 42 


Make predictions about something they are 
reading before reading or as part of reading 


11% 
(1.7%) 
n = 57 


51% 
(2.9%) 
n = 201 


28% 
(2.6%) 
n = 96 




Make generalizations and draw inferences about 
something they have read 


10% 
(1 .8%) 
n = 43 


55% 
(2.9%) 
n = 223 


27% 
(2.6%) 
n = 96 


7% 

(1 .6%) 
n = 25 


Describe the style or structure of a text they have 
read 


28% 
(2.6%) 
n = 118 


53% 
(2.9%) 
n = 201 


15% 
(2.1%) 
n = 51 


4% 
(1 .2%) 
n = 15 



Source: National Survey, Item 21. 

Numbers in parentheses are standard errors. 

Exhibit reads: Of those centers serving elementary school students, 57 percent of centers reported that students were 
asked to read aloud in some reading enrichment activities. 
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Exhibit E-4 

Centers Serving Elementary School Students That Reported Emphases in Math 
Activities, by Type of Content Emphasized and Inclusion Level 



Mathematics Content 


None or 
almost 
none 


Some 

activities 


Most 

activities 


All 

activities 


Number sense 


5% 

(1 .2%) 
n = 18 


36% 
(2.8%) 
n = 138 


41% 
(2.8%) 
n = 156 


18% 
(2.2%) 
n = 73 


Operations (e.g., addition, subtraction, 
multiplication, division) with whole numbers 


2% 
(0.7%) 
n = 8 


28% 
(2.6%) 
n = 109 


46% 
(2.9%) 
n = 172 


24% 
(2.5%) 
n = 97 


Operations with decimals or fractions 


10% 
(1.7%) 
n = 40 


68% 
(2.7%) 
n = 255 


18% 
(2.2%) 
n = 74 


4% 
(1 .0%) 
n = 16 


Measurement 


3% 
(0.8%) 
n = 15 


75% 
(2.5%) 
n = 279 


19% 
(2.3%) 
n = 76 


3% 
(0.9%) 
n = 12 


Algebraic concepts or algebra 


27% 
(2.5%) 
n = 116 


58% 
(2.8%) 
n = 210 


12% 
(1 .9%) 
n = 47 


3% 
(0.9%) 
n = 13 


Geometric concepts or geometry (e.g., area and 
perimeter of shapes) 


17% 
(2.1%) 
n = 83 


70% 
(2.6%) 
n = 253 


11% 
(1 .8%) 
n = 42 


1% 

(0.6%) 
n = 8 


Data analysis (e.g., mean, median, mode) 


31% 
(2.6%) 
n = 130 


62% 
(2.8%) 
n = 223 


6% 
(1 .4%) 
n = 27 


1% 
(0.5%) 
n = 7 



Source: National Survey, Item 24. 

Numbers in parentheses are standard errors. 

Exhibit reads: Of those centers serving elementary school students, 36 percent of centers reported that number sense 
was emphasized in some mathematics enrichment activities. 
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Exhibit E-5 

Centers Serving Elementary School Students That Reported Student Participation in 
Instructional Strategies in Math Activities, by Strategy and Inclusion Level 



Instructional Strategies 


None or 
almost 
none 


Some 

activities 


Most 

activities 


All 

activities 


Explain the reasoning behind how they solved a 
problem 


6% 
(1 .4%) 
n = 26 


47% 
(2.9%) 
n = 184 


32% 
(2.7%) 
n = 123 




Make charts, tables or graphs 


11% 
(1.7%) 
n = 56 


71% 
(2.6%) 
n = 271 


16% 
(2.2%) 
n = 52 


2% 
(0.7%) 
n = 8 


Solve problems in which students have to figure 
out what method to use to solve them 


8% 
(1 .5%) 
n = 37 


50% 
(2.9%) 
n = 196 


34% 
(2.8%) 
n = 124 


8% 
(1 .5%) 
n = 30 


Solve problems that have many possible correct 
answers 


15% 
(2.0%) 
n = 63 


61% 
(2.8%) 
n = 232 


22% 
(2.5%) 
n = 81 


2% 
(0.8%) 
n = 10 


Use computers to practice mathematics 


22% 
(2.4%) 
n = 82 


54% 
(2.9%) 
n = 217 


20% 
(2.3%) 
n = 74 


4% 
(1.1%) 
n = 15 


Write mathematical equations to solve a problem 


17% 
(2.1%) 
n = 75 


53% 
(2.9%) 
n = 205 


26% 
(2.6%) 
n = 85 




Practice basic mathematics facts (e.g., addition, 
subtraction, multiplication and division) 


3% 

(1 .0%) 
n = 12 


29% 
(2.6%) 
n = 116 


46% 
(2.9%) 
n = 171 


22% 
(2.4%) 
n = 89 


Use graphing calculators to solve problems 


70% 
(2.6%) 
n = 270 


27% 
(2.6%) 
n = 102 


2% 
(0.8%) 
n = 12 


0% 

(0.1%) 
n = 1 



Source: National Survey, Item 23. 

Numbers in parentheses are standard errors. 

Exhibit reads: Of those centers serving elementary school students, 47 percent of centers reported that having 
students explain the reasoning behind how they solved a problem was emphasized in some mathematics enrichment 
activities. 



Appendix E: Supplementary Tables for the Surveys 



138 


















































Exhibit E-6 

Centers Serving Elementary School Students That Provided Feedback About Their 
Progress, by Type of Feedback and Center Type 



Feedback 


Overall 


Center Type 


Nonschool- 

based 


School- 

based 


Verbal feedback from center staff on work assigned by 
the school 


87% 
(1 .9%) 
n = 339 


86% 
(3.3%) 
n = 106 


87% 
(2.1%) 
n = 233 


Written feedback from center staff on work assigned by 
the school 


22% 
(2.4%) 
n = 93 


26% 
(4.2%) 
n = 33 


22% 
(2.6%) 
n = 60 


Verbal feedback from center staff on work assigned at 
the center 


72% 
(2.6%) 
n = 279 


74% 
(4.1%) 
n = 88 


72% 
(2.8%) 
n = 191 


Written feedback from staff on work assigned at the 
center 


27% 
(2.5%) 
n = 117 


34% 
(4.5%) 
n = 44 


26% 
(2.7%) 
n = 73 


Verbal feedback from center staff on student behavior 


87% 
(1 .9%) 
n = 336 


87% 
(3.2%) 
n = 105 


87% 
(2.1%) 
n = 231 


Written feedback from center staff on student behavior 


47% 
(2.9%) 
n = 184 


42% 
(4.7%) 
n = 54 


48% 
(3.1%) 
n = 130 


“Points” or rewards for program accomplishments 


50% 
(2.9%) 
n = 207 


65%* 
(4.6%) 
n = 80 


48%* 
(3.1%) 
n = 127 


Certificates or awards for accomplishments 


63% 
(2.8%) 
n = 254 


72% 
(4.4%) 
n = 89 


62% 
(3.0%) 
n = 165 



Source; Focused Survey, Item 18. 

*p<0.05 

Numbers in parentheses are standard errors. 

Exhibit reads: Of those centers serving elementary school students, 87 percent of centers reported providing verbal 
feedback from center staff on work assigned by the school. 
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Exhibit E-7 

Centers That Reported Top-Priority Target Groups, by Target Groups and Center Type 



Top-Priority Groups 


Center Type 


Nonschooi- 

based 


Schooi- 

based 


We have open enrollment for all interested students 


60% 
(14.9%) 
n = 27 


75% 
(5.5%) 
n = 68 


Students who scored “below proficient” on state assessments 


79%* 
(4.4%) 
n = 66 


46%* 
(15.3%) 
n = 19 


Students identified by the school as needing special assistance in 
reading and/or math 


88%* 
(3.6%) 
n = 76 


55%* 
(5.5%) 
n = 20 


Students who receive free or reduced-priced meals 


69% 
(5.2%) 
n = 57 


58% 
(15.0%) 
n = 24 


Students who are recommended by teachers or counselors at the 
feeder schools 


59% 
(5.9%) 
n = 52 


70% 
(12.5%) 
n = 20 


Other 


9% 
(3.1%) 
n = 8 


12% 
(9.8%) 
n = 6 



Source: Focused Survey, Item 1 . 

*p<0.05 

Numbers in parentheses are standard errors. 

Exhibit reads: Of nonschool-based centers, 60 percent reported having open enrollment for all interested students. 
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Exhibit E-8 

Centers That Reported Attendance Requirements, by Grade Level and Attendance 

Requirement 



Attendance Requirements 


Elementary 


Middle 


High 


Attendance is not required; students can drop in and 
out * 


26% 
(2.6%) 
n = 111 


29% 
(2.9%) 
n = 104 


45% 
(5.8%) 
n = 52 


We require students to attend at least 1 day per week * 


3% 
(1 .0%) 
n = 12 


5% 
(1 .4%) 
n = 17 


2% 
(1.7%) 
n = 2 


We require students to attend 2-3 days per week * 


16% 
(2.1%) 
n = 63 




17% 
(4.4%) 
n = 19 


We require students to attend every day we offer 
services * 


41% 
(2.8%) 
n = 147 


31% 
(2.9%) 
n = 97 


22% 
(4.9%) 
n = 25 


Other * 


13% 
(2.0%) 
n = 52 


14% 
(2.2%) 
n = 44 


13% 
(3.7%) 
n = 14 



Source: National Survey, Item 10. 

Numbers in parentheses are standard errors. 

* Significant association (p<0.05) between grade level of school and distribution of attendance items. 

Exhibit reads: Among centers serving elementary school students, 26 percent of centers reported that attendance is 
not required; students can drop in and out. 
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Exhibit E-9 

Centers That Reported That They Required Participation of All Students in Specific 
Activities When Students Were Present, by Type of Activity 



Activities 


Required of 
All 

Students 


Homework help in a group setting 


73% 
(4.6%) 
n = 79 


Tutoring (one-to-one or peer) 


14% 
(3.8%) 
n = 18 


Academic enrichment activities in which reading instruction or practice takes place (not 
tutoring) 


60% 
(5.5%) 
n = 66 


Academic enrichment activities in which mathematics instruction or practice takes 
place (not tutoring) 


58% 
(5.5%) 
n = 63 



Source: Focused Survey, Item 9. 

Numbers in parentheses are standard errors. 

Exhibit reads: Seventy-three percent of centers reported requiring participation in homework help in a group setting 
when students were present. 



Exhibit E-10 

Centers That Reported the Number of Days of Attendance in 2005-06, 
by Grade Level and Days of Attendance 



Duration of Attendance 


Elementary 
N = 75 


Middle 
N = 75 


High 
N = 75 


Fewer than 30 days 


32% 

(3.2%) 


36% 

(2.6%) 


61% 

(1.1%) 


30-59 days 


21% 
(1 .9%) 


28% 

(2.3%) 


9% 

(0.2%) 


60-89 days 


14% 
(1 .6%) 


14% 

(1.1%) 


10% 

(0.2%) 


90-1 99 days 


17% 

(3.3%) 


11% 
(1 .2%) 


14% 

(0.4%) 


120 or more days 


17% 

(2.8%) 


11% 
(1 .0%) 


6% 

(0.2%) 



Source: Focused Survey, Item 8. 

Numbers in parentheses are standard errors. 

Exhibit reads: Thirty-two percent of elementary school students attended fewer than 30 days. 
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Exhibit E-11 

Centers That Reported Consequences for Student Absences, by Consequence Type 



Consequences 


All Centers 


A staff member notifies the student’s parents or caregivers 


58% 
(2.4%) 
n = 292 


A staff member speaks to the student about an absence after a certain number of 
absences 


38% 
(2.4%) 
n = 192 


A staff member speaks to the student about an absence after each absence 


29% 
(2.3%) 
n = 150 


Not applicable; we do not require attendance 


26% 
(2.2%) 
n = 145 


A staff member notifies the student’s school 


13% 
(1.7%) 
n = 67 


The student is restricted from participating in other activities at the center for a period 
of time 


10% 
(1 .5%) 
n = 56 


The student receives “points” or some other kind of penalty that restricts privileges at 
the center 


6% 

(1.1%) 
n = 32 



Source: National Survey, Item 11. 



n = 503 

Numbers in parentheses are standard errors. 

Exhibit reads: Fifty-eight percent of center directors report notifying parents when a student is absent from the 
program. 
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Exhibit E-12 

Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, 

by Type of Staff and Rates 



Pay Rate 


Certified 

Teachers 


Other Staff 


Less than $10 per hour 


1% 
(0.4%) 
n = 12 


29% 
(2.3%) 
n = 162 


$10-$15 per hour 


11% 
(1 .5%) 
n = 63 


52% 
(2.5%) 
n = 256 


$15.01-$20 per hour 


27% 
(2.3%) 
n = 129 


10% 
(1 .5%) 
n = 44 


$20.01-$25 per hour 


27% 
(2.2%) 
n = 128 


3% 
(0.8%) 
n = 12 


$25.01-$30 per hour 


20% 
(2.0%) 
n = 89 


1% 
(0.5%) 
n = 4 


More than $30 per hour 


10% 
(1 .5%) 
n = 41 


0% 
(0.3%) 
n = 2 


Don’t know 


6% 

(1.1%) 
n = 33 


5% 

(1 .2%) 
n = 21 



Source: National Survey, Item 27. 

Numbers in parentheses are standard errors. 

Exhibit reads: Among all centers, 1 percent reported that certified teachers leading instructional activities were paid 
less than $10 an hour. 
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Exhibit E-13 

Staff Who Led Instructional Activities at Centers, by Hours Worked per Week 



Hours Worked 


All Centers 


Full-time (35 hours or more per week) 


10% 
(2.4%) 
n = 50 


Half-time (20-34 hours per week) 


14% 
(2.8%) 
n = 51 


Part-time (10-19 hours per week) 


44% 
(4.3%) 
n = 140 


Occasionally (less than 10 hours per week) 


32% 
(4.4%) 
n = 110 



Source: Focused Telephone Survey, Items 26, 32 and 38. 

Numbers in parentheses are standard errors. 

Exhibit reads: Ten percent of centers reported staff leading instructional activities worked full-time (35 hours or 
more per week). 



Exhibit E-14 

Centers That Reported Offering Benefits to Full- and Part-Time Staff Who Led 
Instructional Activities, by Benefit Offered and Staff Status 



Benefits 


Full-Time 

Staff 


Part-Time 

Staff 




40% 


11% 


Paid time off for vacation and sick leave 


(2.4%) 


(1 .5%) 




n = 244 


n = 66 




43% 


8% 


Health insurance 


(2.4%) 


(1 .3%) 




n = 248 


CO 

CO 

II 

c 




41% 


14% 


Retirement savings plan 


(2.4%) 






n = 234 






12% 


4% 


Tuition reimbursement 


(1 .6%) 






n = 65 





Source: National Survey, Item 28. 

Numbers in parentheses are standard errors. 

Exhibit reads: Eorty percent of centers reported that full-time staff leading instructional activities were offered paid 
time off for vacation and sick leave. 
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Exhibit E-15 

Centers That Reported Reasons for Staff Departures, 
by Reason for Departure and Center Type 



Reasons for Leaving 


Center Type 


Nonschool- 

based 


School- 

based 


Low pay 


18%* 
(3.4%) 
n = 27 


8%* 
(1 .5%) 
n = 32 


Lack of benefits 


44%* 
(4.3%) 
n = 66 


25%* 
(2.3%) 
n = 92 


Inability to offer a full-time position 


24% 
(3.5%) 
n = 40 


22% 
(2.2%) 
n = 80 


Completion/graduation from a program (e.g., AmeriCorps) or from school 


42% 
(4.3%) 
n = 64 


49% 
(2.7%) 
n = 179 


Don’t know 


9% 
(2.9%) 
n = 10 


13% 
(1 .8%) 
n = 48 


We have not had staff leave our program 


28%* 
(3.9%) 
n = 43 


12%* 
(1.7%) 
n = 46 



Source: National Survey, Item 30b. 

*p<0.05 

Numbers in parentheses are standard errors. 

Exhibit reads: Among school-based centers, 8 percent reported low pay as the primary reason that staff leading 
instructional activities left. 
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Exhibit E16 

Centers That Reported That They Offered Paid Professional Development Opportunities 
for Full- and Part-Time Staff Who Led Instructional Activities, 
by Staff Status and Type of Development 



Professional Development 


Full-Time 

Staff 


Part-Time 

Staff 


Paid attendance at conferences 


53% 
(2.5%) 
n = 304 


39% 
(2.4%) 
n = 223 


Paid training or professional development 


62% 
(2.4%) 
n = 338 


55% 
(2.5%) 
n = 307 



Source: National Survey Item 28. 

Numbers in parentheses are standard errors. 

Exhibit reads: Fifty-three percent of centers reported that paid attendance at conferences was a professional 
development opportunity for full-time staff leading instructional activities. 



Exhibit E-17 

Centers That Reported That State Assessments Data Was Available for Use in Tracking 
Student Outcomes, by Subject Tested and Center Type 



State Assessment Results for School 


Center Type 


Nonschool- 

based 


School- 

based 


Reading 


53%* 
(4.3%) 
n = 79 


73%* 
(2.4%) 
n = 268 


Mathematics 


52%* 
(4.3%) 
n = 77 


71 %* 
(2.4%) 
n = 261 



Source: National Survey, Item 13. 

*p<0.05 

Numbers in parentheses are standard errors. 

Exhibit reads: Among school-based centers, 73 percent reported that centers could track student outcomes in reading 
using state assessment results. 
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Exhibit E-18 

Centers That Reported Student Assessment Sources, 
by Type and Frequency of Assessment 



Benefits 


Not used 


1-2 
times 
per year 


1-2 

times 

per 

month 


Weekiy 


Aimost 

daiiy 


Informal staff observations 


26% 
(2.2%) 
n = 124 


12% 
(1.7%) 
n = 54 


14% 
(1.7%) 
n = 76 


20% 
(2.0%) 
n = 92 


28% 
(2.3%) 
n = 149 


Written reports from teachers 


36% 
(2.5%) 
n = 161 


34% 
(2.4%) 
n = 165 


19% 
(2.0%) 
n = 97 


9% 
(1 .4%) 
n = 39 


3% 
(0.8%) 
n = 12 


Verbal reports from teachers 


26% 
(2.2%) 
n = 133 


11% 
(1 .6%) 
n = 61 


23% 
(2.1%) 
n = 118 


26% 
(2.2%) 
n = 123 


13% 
(1.7%) 
n = 60 


Results from tests administered by 
center staff 


58% 
(2.5%) 
n = 260 


24% 
(2.2%) 
n = 123 


11% 
(1 .6%) 
n = 62 


5% 

(1.1%) 
n = 24 


1% 
(0.5%) 
n = 11 


Results from tests administered at 
students’ schools 


31% 
(2.3%) 
n = 155 


47% 
(2.5%) 
n = 242 


13% 
(1.7%) 
n = 62 


7% 
(1 .3%) 
n = 30 


1% 

(0.6%) 
n = 7 


Student self-report or self-assessment 


50% 
(2.6%) 
n = 235 


22% 
(2.1%) 
n = 104 


10% 
(1 .6%) 
n = 51 


9% 
(1 .5%) 
n = 47 


8% 
(1 .4%) 
n = 39 


Informal discussions with family 
members 


32% 
(2.4%) 
n = 151 


22% 
(2.1%) 
n = 106 


22% 
(2.1%) 
n = 104 


16% 
(1 .8%) 
n = 88 


7% 
(1 .3%) 
n = 38 


Other 


87% 
(2.8%) 
n = 136 


7% 

(2.2%) 
n = 14 


4% 
(1.7%) 
n = 7 


0% 

(0.2%) 
n = 2 


2% 
(0.9%) 
n = 4 



Source: National Survey, Item 15. 

Numbers in parentheses are standard errors. 

Exhibit reads: Twenty-six percent of centers reported informal staff observations are not used to assess students’ 
academic growth. 
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Exhibit E-19 

Centers That Reported Uses of Student Assessment Data, by Type of Use 



Indicators 


All Centers 


We do not use these data in our program 


1% 
(0.5%) 
n = 7 


To identify needs for new program offerings 


59% 
(2.4%) 
n = 310 


To make improvements to practices within existing program offerings 


69% 
(2.3%) 
n = 360 


To assign students to particular activities 


48% 
(2.5%) 
n = 258 


To assign students to particular staff who are qualified to help students in identified 
academic areas of need 


54% 
(2.5%) 
n = 294 


To assign students to particular groups or to older youth in the program 


27% 
(2.1%) 
n = 155 


To evaluate the success of the program 


66% 
(2.4%) 
n = 345 



Source; National Survey, Item 16. 

Numbers in parentheses are standard errors. 

Exhibit reads; One percent of centers reported not using assessment data in their program. 
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Exhibit E-20 

Centers That Reported Challenges In Implementing High-Quality Programming, by Type of 

Challenge and Center Type 



Challenges 


Center Type 


Nonschool- 

based 


School- 

based 


Lack of information about the school-day curriculum 


40% 
(4.3%) 
n = 57 


31% 
(2.5%) 
n = 113 


School staff not responsive to requests from the center 


39%* 
(4.3%) 
n = 55 


29%* 
(2.4%) 
n = 108 


Lack of information about students' academic needs 


36%* 
(4.2%) 
n = 54 


20%* 
(2.1%) 
n = 77 



Source: National Survey, Item 31. 

*p<0.05 

Numbers in parentheses are standard errors. 

Exhibit reads: Among nonschool-based centers, 40 percent reported lack of information about the school-day 
curriculum as a challenge in implementing high-quality programming. 
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Appendix F 

21st CCLC Quality Study — Quality Observation Instrument 




21st Century Community Learning Centers 
Program Quality Study - Quality Observation Instrument 



Center 

number: 



Observer Co-observer Observation 

initials: initials: number: 



Room 

number: Date: 

TTTi [n/[n/[n 

mm d d y y 



ACTIVITY SUMMARY 



ACTIVITY NAME: 



ACTIVITY DESCRIPTION: 



ACTIVITY TYPE (Mark all that apply.) 

Q Homework 

Q T utoring (one-on-one, small group) 

□ Language arts enrichment activities (other than homework) 

□ Math enrichment activities (other than homework) 

□ Computer-based skill building 

□ Computer instruction (tools, Internet, research) 

Q Teacher-assigned learning games (dominos, chess, etc.) 

□ Arts activities (crafts, visual arts, graphics, dance, music, drama) 

□ Sports/Recreation 

□ Open, unstructured time (e g , table games, Internet, free play) 

Other: I I 



TYPE OF SPACE (Mark one.) 

Q Classroom 

□ Gym 

Q Computer Lab 

□ Library 

□ Cafeteria 

□ Auditorium 

□ Art Room 

□ Music Room 

□ Hallway 

□ Outside Playground 

□ Other: I 



SKILLS TARGETED (Mark one.) 

□ Language arts 

□ Mathematics/numeracy 

□ Mixed academic skills (e g., homework) 

□ Library 

□ Artistic 

□ Physical/athletic 

□ Interpersonal/conflict resolution 

□ No specific skill 

□ Other I 



PARTICIPATION TYPE (Mark one.) 

□ By age/grade 

□ By interest (child’s choice) 

□ By selected group (staff determined) 

□ All attendees (in the project) 

GRADE LEVELS: (Mark all that apply.) 

Q Pre-K QK □ 1st Q2nd Q3rd □ 4th QSth 

□ 6th □Tth □Sth noth □lOth □ 11th □ 12th 

□ Other: I I 



TOTAL STAFF (Number) 

Degreed specialist 

Middle/High College student or other Certified 
school student: or young adult: professional: teacher: 

m m m m 

TOTAL STUDENTS (Number): | | | | 



Other 



adult; Other: 


(Describe:) 


m m 
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DOMAIN/INDICATOR RATINGS 



RATINGS: 

1 = Did not occur. No evidence of the behaviors described was observed. 

2 = Uncharacteristic. The behavior described by the indicator was not characteristic of the activity; 
exemplars were observed infrequently or exhibited by only one staff or student. 

3 = Somewhat characteristic. There is good evidence that the behavior described by the indicator is 
present, but it is not pervasive and continuous. 

4 = Highiy characteristic. The behavior described by the indicator is highly evident. Exemplars are 
observed frequently and consistently throughout the observation and with the majority of students 



RELATiONSHiP BUILDING: With all students. STAFF 12 3 4 

a. Use positive behavior management techniques that allow students to acoomplish the 
aotivity's objectives. They set appropriate limits and set clear behavior standards, and these are 

appropriate to the age of the students and the activity type. If it is necessary to discipline □ □ □ □ 

students, staff do so in a firm manner, without unnecessary accusations, threats, or anger. 

There is no evidence of disciplinary problems. 

b. Show positive affect toward students. Staff tone is caring and friendly; they use positive 

language, smile, laugh, or share good-natured jokes. They refrain from threats, cutting D D D D 

sarcasm, or harsh criticism. If no verbal interaction is necessary, staff demonstrate a positive 
and caring affect toward students. 

c. Attentively listen to and/or observe students. Staff look at students when they speak, 

and acknowledge what students have said by responding verbally or nonverbally. They pay □ □ □ □ 

attention to students as they complete tasks. 



d. Encourage students to share their ideas, opinions and concerns. Staff actively elicit 
students ideas, opinions and concerns through discussion and/or writing. 



□ 



□ 



□ 



□ 



SKILL BUILDING AND MASTERY: STAFF 12 3 4 

e. Ask students to expand upon their answers and ideas. Staff encourage students to 
explain their answers, evidence, or conclusions. They may ask students "why", "how" and "if 

questions to get them to better clarify, articulate, or explain their thoughts/ideas. Or, students □ □ □ □ 

spontaneously explain/expand answers to peers as a result of the way the activity is organized 
and without adult prompting. 



f. Assist students without taking control. Staff refrain from taking over a task or doing 
something on behalf of the students. Staff may coach, demonstrate, or employ 
scaffolding techniques that help students to gain a better understanding of a concept or 
complete an action on their own. 



□ □ □ □ 



g. Employ two or more teaching strategies. In order to engage students and reach 
those with different learning styles, staff diversify instructional strategies, which may 
include the use of two or more of the following: direct instruction, coaching, modeling, 
demonstrating, project-based work, use of manipulatives or experiments, or others. 



□ □ □ □ 



h. Challenge students to move beyond their current level of competency. Staff 

provide feedback that encourages, motivates, or challenges students. Staff encourage □ □ □ □ 

students to push themselves intellectually, creatively, and/or physically. 
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DOMAIN/INDICATOR RATINGS 



RATINGS 

1 = did not occur 2 = uncharacteristic; 3 = somewhat characteristic; 4 = highly characteristic; 



SKILL BUILDING AND MASTERY: STAFF (Continued) 


1 


2 


3 


4 


i. Communicate goals, purpose, expectations. Staff explain the goals and purpose of 
what students are doing and/or what they expect them to accomplish. The goals and 
expectations of the activity may also be implicit if students are clearly on task without 
staff direction. 


□ 


□ 


□ 


□ 


j. Recognize students' efforts and accomplishment. Staff explicitly or implicitly 
acknowledge students' participation and progress through praise, encouragement, 
non-verbal signals (nodding, smiling, etc.). 


□ 


□ 


□ 


□ 


k. Plan for/ask students to work together. Staff structure activities to encourage 
students to work together to solve problems, and accomplish tasks In the case of 
staff-assigned teams for competitive games and sports, staff encourage students to 
collaborate and plan. 


□ 


□ 


□ 


□ 


CONTENT AND STRUCTURE: ACTIVITY 


1 


2 


3 


4 


1. Is well organized. Activity has clear (implicitly- or explicitly-stated) goals/objectives; 
there is evidence of a clear plan and process(es), and tasks can be conducted in the 
time frame available If special materials are needed, they are prepared and available. 


□ 


□ 


□ 


n 


m. Challenges students intellectually, creatively, and/or physically. Activity's level of 
challenge is not so difficult that students have trouble participating successfully and not 
so easy that students complete tasks quickly and become restless/disengaged. 


□ 


□ 


□ 


□ 


n. Involves a progression or the practice of skills. Activity involves learning or 
practicing of skills needed to complete tasks or to participate. If a long-term project, 
students build on skills or techniques previously learned. 


□ 


□ 


□ 


□ 


o. Requires analytic thinking. Activity calls on students to think about and solve 
meaningful problems and/or consider multiple activities or strategies to accomplish a 
task. 


□ 


n 


□ 


□ 


TASK ORIENTATION: STUDENTS 










p. Are on-task. Students are focused, attentive, and not easily distracted from the 
task/project. They follow along with the staff and/or follow directions to carry on an 
individual or group task. Noise level and student interaction can be high if they are 
engaged in the task expected of them. 


□ 


□ 


□ 


□ 



ENVIRONMENTAL CONTEXT 



At the end of the observation, complete the following: 


No 


Yes 


Is the level of adult supervision appropriate to activity and age group? 


□ 


□ 


Is the work space conducive to the activity type? 


□ 


□ 


Are students working on a skill-building activity? 


□ 


□ 


Are special materials needed for this activity? 


□ 


□ 


If so, are materials in sufficient supply? 


□ 


□ 
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ACADEMIC QUALITY RATING 
Literacy, Mathematics, and Technology 

RATINGS: 

Not at all. Activity did not occur at ail. 

Embedded. Content/feature is practiced through the activity, although it may not be explicitly stated. 

Explicit. Content/feature is addressed directly by the assignment or activity, through staff statements 
or written instructions. 




LITERACY FEATURES 



Q If this activity/lesson is NOT literacy focused, mark the box and move to the next section. 
If it is literacy-focused, mark the appropriate response in each row. 


in this literacy activity, students: 


Not 
at all 


Embedded 


Explicit 


a. Read aloud to peers or staff 


□ 


□ 


□ 


b. Talk with peers about what they have read 


□ 


□ 


□ 


c. Work on open-ended creative writing 


□ 


□ 


□ 


d. Work in a reading workbook or on a worksheet 


□ 


□ 


□ 


e. Choose their own books to read 


□ 


□ 


□ 


f Do a group activity or project about what they have read 


□ 


□ 


□ 


g. Work on understanding new words 


□ 


□ 


□ 


h. Write answers to structured questions about something they have read 


□ 


□ 


□ 


i. Make predictions about something they are reading before reading 
or as part of reading 


□ 


□ 


□ 


j. Make generalizations and draw inferences about something they 
have read 


□ 


□ 


□ 


k. Describe the style or structure of a text they have read 


□ 


□ 


□ 


1. Discuss different interpretations of what they have read 


□ 


□ 


□ 


m. Watch videos/DVDs or listen to music 


□ 


□ 


□ 



In this literacy lesson, the teacher or tutor Not 



develops students' knowledge/understanding of: at all Embedded Explicit 
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MATHEMATICS FEATURES 

Q If this activity/lesson is NOT mathematics focused, mark the box and move to the next section. 
If it is mathematics-focused, mark the appropriate response in each row. 




in this mathematics activity/lesson, students: 


Not 
at all 


Embedded 


Explicit 


a. Explain the reasoning behind how they solved a problem 


□ 


□ 


□ 




b Make charts, tables, or graphs 


□ 


□ 


□ 




c. Solve problems in which they figure out what method to use 


□ 


□ 


□ 




d Solve problems that have many possible correct answers 


□ 


□ 


□ 




e. Use computers or calculators to practice basic math skills 


□ 


□ 


□ 




f Write mathematical equations to solve problems 


□ 


□ 


□ 




g. Practice basic math facts (e.g., addition, subtraction, multiplication, i-i 

and division) ^ 


□ 


□ 




h. Use graphing calculators to solve complex math problems 


□ 


□ 


□ 




In this mathematics lesson, the teacher or 
tutor develops among students: 


Not 
at all 


Embedded 


Explicit 


a. Number sense 


□ 


□ 


□ 




b. Operations with whole numbers (e.g., addition, subtraction, 
multiplication, division) 


□ 


□ 


□ 




c. Operations with decimals or fractions 


□ 


□ 


□ 




d. Measurement 


□ 


□ 


□ 




e. Algebraic concepts (e g., solving equations) 


□ 


□ 


□ 




f. Geometric concepts (e g., area and perimeter of shapes) 


□ 


□ 


□ 




g. Data analysis (e g., mean, medium, mode) 


□ 


□ 


□ 




USE OF TECHNOLOGY 










□ If computers are NOT used in this activity/lesson, mark the box and move to the next section. 


If computers are used, How many computers are being used during this observation? T 


n 




Which of the following tools do students work 

with on the computer? Mark the appropriate Not at 

response in each row. all 


By fewer than 
half the 
students 


By half or By almost all 
more of the of the 

students students 


a. Internet search tools (e g , Google, Yahoo!) □ 


□ 


□ 




□ 


b Spreadsheet program (e g.. Excel) □ 


□ 


□ 




□ 


c. Word processing program (e g , Microsoft Word) □ 


□ 


□ 




□ 


d. Presentation software (e g , PowerPoint) D 


□ 


□ 




□ 


e Drill and practice software for reading (e g., i-i 

Destination Reading, READ 180) ^ 


□ 


□ 




□ 


f. Drill and practice software for mathematics 

(e g . Achieve Now, SuccessMaker) LJ 


□ 


□ 




□ 



g. Other: 
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OBSERVATION SUMMARY 



Briefly provide illustrative examples of interactions, activities, comments (by students and staff), and 
behaviors observed within each domain. Add any additionai cross-cutting comments on the overall 
quality of the lesson and the staff-student interactions under "Observer Comments." 



Domain Description 


Illustrative Examples 


1. Activities promote iearning. Activities advance 
students' knowledge, skills, understanding, and 
interest. Staff provide activities and/or instruction, 
support an environment in which engaged learning 
occurs (use the Activity Checklist as the basis of 
this summary) 




2. Staff-directed relationships. Staff provide 

guidance and emotional support; they take interest 
in the students and their ideas 




3. Skill building and mastery. Staff strategies are 
geared towards encouraging students to push 
beyond their present level of competency 
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OBSERVATION SUMMARY, Continued 



Briefly provide iliustrative examples of interactions, activities, comments (by students and staff), and 
behaviors observed within each domain. Add any additional cross-cutting comments on the overali 
quaiity of the iesson and the staff-student interactions under "Observer Comments." 



Domain Description 


Illustrative Examples 


4. Activity content and structure. Activities are 
planned and well organized; challenge level is 
appropriate to age; there are opportunities for 
problem solving 




5. Task Orientation. Task orientation is defined by 
involvement with the expected activity for the 
period, whether it involves interaction, students 
working on their own, or response to a 
staff-directed lesson. Describe the level of task 
involvement of participants during this observation 
period. 




6. Technology. Describe the type of technology 
resources available for use during this activity, 
what teachers and students are doing with the 
technology, how many students were using the 
technology, and their level of engagement with it. 





H 9 9 9 9 9 



Page 7 of 8 



39078 



hT 




Appendix F: 21st CCLC Quality Study - Quality Qbservation Instrument 



159 





OBSERVER COMMENTS 
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